Export Messages

As short side project I created an app that exports messages from the webOS 2.x and 3.x message database and writes them into a text file.

There were some people on the webosnation forum who asked for a way to get their messages into Excel: http://forums.webosnation.com/hp-pre-3/311205-sms-mms-transfer.html

Because they had some thousand messages my advices with command line stuff did not help them so much, so I figured I write a small app. The issue with the app is, that it needs access to the message db, which apps usually don’t get in webOS. To circumvent that issue, there is  either a command line that you have to execute or you can install a patch. The app is published in the preware feeds: http://www.webosnation.com/export-messages

Also the code for the app is on github: https://github.com/Garfonso/ExportMessages

The patch has been submitted to the patch portal. But you can also download it from github: https://raw.github.com/Garfonso/ExportMessages/master/permission.patch

I hope that helps some people.

Contact Sync

Just a short status update (I really need to configure Poster for this blog. ;)).

I implemented and published a first experimental contact sync. The contact sync currently has multiple issues:

  • Character encoding issue (for example umlauts)
  • Photos do not get synced at all
  • Some servers (for example eGroupware) behave very strange. I believe this is, because they don’t know my app
  • Contacts don’t show up on the TouchPad at all

The TouchPad issue is quite strange. The contacts are correctly stored in the database… the same code works fine on the phones. But on the TouchPad the contacts app and also Just Type won’t show the contacts at all. I need to investigate that further…

Currently I advise you to be very careful, if you use contact sync… maybe first try One-Way-From-Server sync.

BTW: I fixed an issue that made the app forget One-Way-From-Server (or similar) if the server requested a slow sync and made it switch to “Two-Way” sync. This is now fixed and slow sync will switch back to One-Way-From-Server correctly.

Status Update

I recon that I did not write much here for quite some time. I think it’s time for a status update.

First, the most important thing: I’m still actively working on this and I’m also still using this on my webOS Phones and TouchPads. 🙂

Now, what works:

  • Works on webOS 2.2.4 Phones and webOS 3.0.5 tablets (2.1.1/2.1.2 works, but has some quirks sometimes. Need someone to test that version)
  • Installation via Precentral Feeds (i.e. via Preware on the device)
  • Calendar, no Contacts yet, still, sadly… 🙁
  • Calendar sync from the stock calendar app
  • Setup accounts from stock account app (Phones only)
  • Setup and modify accounts from SyncML app
  • Periodic sync of calendar (like every hour or every day, configurable from app or during creation. FYI: google calendar syncs every 20min). Sync only happens if there is a “fair” internet connection

Most probably I, again, did miss something… but I can tell you that calendar stuff is working very well for me with eGroupware over here. I am testing some other servers during development, but of course not so very exhaustive… If you use some server that supports SyncML for your calendar (like funambol, most webbased groupwares or even evolution with syncevolution on linux desktops), feel free to try this SyncML Synergy Connector (and App) and if something does not work, give feedback to me. Either here, via E-Mail or via the webOS nation forums. I’ll be listening. 🙂

node.js Service

Three days of hard work, but now the node.js service is completely implemented. The most parts of my js libraries could be transfered without many changes from the app to the node.js service. 🙂

Now the app merly is a gateway to the service, and we  have done the first important step towards automatic synchronization in the background.

Sadly though, I renamed  the service in early april. That, together with this nasty webOS bug hindered my service from working. Luckily only devices with the old, not working service with the old name installed are affected and I did the rename before I submitted the app to webos nation. So most people should be fine. 🙂

For the rest here are the instructions how to repair it:

  1. [Optional, step 3 might delte all data on the device] Install version 0.0.16 (get it from github) and do a sync of your data
  2. Install 0.1.1
  3. Execute sh /media/cryptofs/apps/usr/palm/applications/info.mobo.syncml.client/repairDBPermission.sh on a command line on the device, use novaterm or webosquickinstall or a ssh connection
  4. Run the app
  5. Maybe you need to configure the account again
  6. Anyway, run a sync with “Refresh-from-server” set.

The biggest improvement for the user right now is much cleaner logs. The majority of the logging is now done in the background to the file  /media/internal/.info.mobo.syncml.log. In the app only some (hopefully) useful messages show up to give an impression of the status of the sync and what is working and what not. 🙂
Also the log is now send as attachement, which helps the webOS E-Mail app a lot. People can also get it via usb mode. It’s a great change, I think.

So… what next? Automatic background syncs? Or contacts?

Precentral Feeds

This weekend I decided to publish the current version of the app to the Precentral Feeds (or webosnation now?). If I did not make a mistake during the process the app should appear in the feed soon.

Also I cleaned up some more things. Now the sync automatically starts in the sync scene (a by product of the protection against to early start sync presses).

I hope I’ll get some test results from that. For me this works fine for months now.

Service can load library

It’s been quite sometime since the last update here and I must admit development also was on hold a bit. The current version still is working quite good for me, but the UI is not intuitive… I did a little change to the app version that it selects the first saved account automatically on start.

But still it will show statistics during sync, that are not working (so the numbers in the top of the app are not updated, they will always stay 0). In the logs there are some statistics, yes… But that’s not really intuitive.

One good thing for that is, that the main reason for the hold of development, my issues I had to load the syncml library I wrote in the node.js service are resolved. That is what I achieved this weekend. So I will be able to write a node.js service that does the synchronization. From what I understand that can be hooked (somehow? It’s undocumented as it seems :() into the sync of accounts, so that a SyncML account can act like a Google calendar or similar. What is still missing are the service assistants.

Additionally the service should help with contact sync implementation. A new Version can be downloaded here.

SyncML Status 18.1.

I’m making some progress with the SyncML client.

Todos:

  • add try/catch to all callbacks to receive notions of errors that happen. Especially look at iCal, eventCallbacks and SyncML. Maybe als SyncMLMessage.
  • write to O3SIS and ask if they have an idea why their server (used by O2) does not accept my device information.
  • work on js service implementation
  • implement message retry mechanism, sometimes I only receive an empty response… 🙁
  • test on funambol server and on http://www.o-sync.com
  • supply the  type of iCal to iCal.js, i.e. if it’s 1.0 = x-vcalendar or 2.0 = vcalendar. Do what you can for x-vcalendar conversion (not so easy).
  • save global event id in externalId

Issues to look at:

  • O2 server does not accept allDay events from the device (thought I had fixed that)
  • look in “quoted_printable_decode/encode” to try to solve umlaut issues and similar
  • does the webos calendar without patches allow to choose the colour of an calendar?
  • eGroupware did send me duplicates of allDay events during a slow sync. Those were not duplicated on the server which means either I did not send them (or what I send was faulty) or…? Look into that. Maybe this was because of the ongoing timezone issue?
  • see if funambol needs sync to “configuration” datastore for credentials checking or if my empty SyncMLMessage is sufficient… Example for funambol configuration sync:
    <Alert><CmdID>1</CmdID><Data>200</Data><Item><Target><LocURI>configuration</LocURI></Target><Source><LocURI>config</LocURI></Source><Meta><Anchor xmlns=”syncml:metinf”><Next>1268079974281</Next></Anchor></Meta></Item></Alert>
    URL: https://core.forge.funambol.org/ds/viewMessage.do?dsForumId=405&dsMessageId=94713

Thoughts about contact sync:
Contacts Service has readVCard => can I call that?

Maybe need to write vCard parser myself. Does not seem more complicated than iCal parser. Some ressources:
http://en.wikipedia.org/wiki/VCard#Properties
https://tools.ietf.org/html/rfc6350
maybe also:
https://tools.ietf.org/html/rfc4770
or extensions mentioned in wikipedia.

This is also interesting: frameworks/contacts/…/vCard/vCard.js
Has importer, here I can set importToAccountId to my accountId and let it do all the work. There also is a importToContactSetId. Maybe that is useful to implement categories? Keep in mind for later… 😉
Let it return an array of contacts to the app, so I can do the add’s myself and supply the local ids to the server mapping. Also this seems necessary for updates instead of adds.

Interesting also: VCardImporter => processOneContact creates hashes to check for duplicates. Maybe that is a good thing to have in case something goes wrong?

Look what is the same for contacts and events and move that into common js files to make the single files smaller. It’s getting quite messy already. 🙁

SyncML as Synergy Connector

Since a long time now I am using an eGroupware Server at home as my personal cloud. I tried some hacks to use it with webos, but none really worked out as I liked it. Most lead to data corruption and I always was required to leave my valuable data (what I’m doing everyday!!!) on big company servers like google.

SyncML on webOs 1.4.5

So I started a project to implement a SyncML Client for webOs, because SyncML worked good for me with Windows Mobile 5-6.5. I first tried to implement something based on some SyncML libraries. I managed to combile funambol for webOs and build a client app around that. It worked good for calendar syncs. But it never made me really happy. You need to manually fire up the app and start a sync and the sync is quite slow. There are some issues that reside deep inside the funambol library and in the iCal parsing I needed to do. Then this all was for webOs 1.4.5, which mostly was not  usable for webOs 2.x anymore that my new phones are all using (only my old Pixi Plus is on 1.4.5, still and comes with me as second phone just for calendar syncs since some time now). I won’t work on SyncML for webOs 1.4.5 anymore, but you can get the code and an ipkg here.

SyncML on webOs 2.x

As I said, my 1.4.5 version won’t work on webOs 2.x. I fixed it to basically run on 2.x (you can view that here) but never managed to fix the issues I had. There were bad conversion issues from iCal to the new calendar event structure in webOs 2.x.
But the worst issue with that approach was, that it would never be able to work as new and cute synergy connector. So I restarted from scratch and implemented SyncML and a new iCal parser in pure JavaScript (which made the libraries much more flexible and smaller and I understand them and can fix issues. :)). I (ab)used the last holydays to finish  those libraries and today debugged my fingers off, but now I am happy to announce that I managed to get something that syncs with my eGroupware Server via SyncML. There are still some missunderstandings between the two which my iCal Parser tries to circumvent. But it works.
It’s far from finished and still has the old app interface, but it is on a good way forward. You can have a look at it here.

My list of Todo’s

  • Build synergy integration
  • Build service for that
  • Rebuild App to use the service (for easier debugging)
  • Test the service
  • Modify account management  and sync management for better integration of calendar and contacts to add more datastores
  • Implement contacts sync (will require new or extended parser)
  • Modify app to view service logs and allow to send them per E-Mail
  • Optimize old eventCallback structure to be more database optimized and collect all adds, updates and deletes to do them in one batch.
  • Try to respect server max msg size by only adding like 10 objects to a message (currently sends all in one message and has reached like 20kb which is way over the server advertised limit of 10kb. Luckily that does not matter much to the server, as it seems).