‘Cannot find phone’ error in software updater

I’m reflashing one of our N95′s to take it into use. But for some reason the Nokia Software Updater wasn’t finding the phone even after all the drivers were installed, the computer rebooted, the phone reconnected etc.

Turned out the connectivity driver had picked a COM port for the USB connection that was already in use.

So to fix the error, you just have to connect the phone, go to Device Manager, go to the N95 COM port with a warning icon, properties, change the COM port to a reasonable one, disconnect, connect again and voila – the updater finds the phone.

Mebbe this is useful to somebody out there.

Posted in Symbian | 7 Comments

I will defend “Exploring privacy for ubiquitous computing: Tools, methods and experiments” on May 29th

At 12 o’clock in the University of Helsinki Main building Auditorium XII.

Introduction at http://mikie.iki.fi/dissertation/intro.pdf.

Feel free to join in on my public humiliation.

Posted in Uncategorized | 2 Comments

TCP is not reliable, it’s just ordered

I gave a talk last Sunday in the Free and Open Source Developers’ European Meeting (FOSDEM) 2007 on how Jaiku uses Jabber, aka XMPP, (Slides with notes, even if I don’t usually do that).

I realized that there was this undertone in my talk that TCP is somehow broken by GPRS, since packets get acked even if they are not received by the phone and that the connection breaks a lot and has a large latency. But that’s of course not true: TCP isn’t being _broken_ by this, just our mistaken view of what TCP is.

TCP tends to be describe as reliable: RFC 793 introduces it as a “highly reliable host-to-host protocol”; Wikipedia claims that it “guarantees reliable [...] delivery” and man tcp calls it “reliable” as well.

The trick, is that the “reliability” this documents talk about is quite different with what we tend to think of as reliable. TCP’s reliability means in-order delivery and integrity (no bits flipped). What people often think when they think of reliable is some vague idea of “data getting to the other end”. Which of course isn’t the case.

If you want reliability, you 0) give your data a unique identifier, 1) store persistently the data you are going to send, 2) send it to the other side 3) store the data persistently on the other side, 4) send an acknowledgement to the sender, 5) delete on the sender. TCP does something like this, but it forgets ‘persistently’, since it’s just in the OS buffers which get thrown away if you close the socket, your program, or the machine.

So, guys and gals, TCP is just ordered: if you send x, y and z the received will never receive just x and z. But that’s all. If you want “reliability” you have to do what I described above on the application level.

Posted in CS | 10 Comments

Forum Nokia FUD

Recently, I and some other people participated in a Forum Nokia Discussions thread on how to reboot the phone gracefully. I pointed out the undocumented SysStartup::ShutdownAndRestart() function with some example code, which was promptly removed. This sparked the following discussion (I’m just reposting my own text with enough context from others to situate it).

[QUOTE=ltomuta]This forum is moderated and every now and then some posts may have to be “removed” at least until they are properly analyzed. The code posted here uses internal APIs which you are not authorized to use and/or distribute.[/QUOTE]

Since this is Nokia’s Forum, they are of course allowed to remove anything they want, with mainly the risk of people not posting anything useful anymore (which is mainly the case anyway). But since they do not have any other authority over the code, it’s futile too: just search google for the relevant terms (although I admit that most of the users on these forums do not seem capable of doing that).

The claim that there can be code that ‘uses internal APIs which you are not authorized to use and/or distribute’ is in its generality pure FUD.

You (I in this case) may be limited in distributing the header by copyright. A partial header from an SDK is with thousands of headers very unlikely to be protected by copyright.

Should the poster have signed some kind of agreement with Nokia where they have agreed to protection beyond copyright, such as an NDA, the poster may not actually have the right to post such information. This is not the case here (I have signed an NDA with Nokia, but it does not cover information I have received through other channels or before the agreement).

If the poster is within their rights to post a header snippet there is no way somebody could not be ‘authorized to use’ the API, unless they have also signed an agreement with Nokia where they expressedly give up any such right. I don’t think anybody has.

The header in question is also trivially generatable from the SYSUTIL.LIB library file, with only the parameter values to be deciphered. These parameter values are (drumroll, please): 100 and any UID. This information is definitely not protectable under copyright.
Reverse engineering for interoperability is expressedly permitted in the EU, and many other legislations.

With 3rd ed, you are of course of at the mercy of Nokia as to whether you can get an app that uses this API signed. That’s the life of a sharecropper.

Mika Raento
Forum Nokia Champion

[QUOTE=ltomuta]Thanks for pointing me to that thread Paul. I took the posts out but I have to say that I am quite disappointed seeing who are those demonstrating that advanced knowledge is not always good friend with wisdom.[/QUOTE]

And just in case anybody still feels like they shouldn’t be rebooting the phone: the only reason we automatically reboot the phone is that basically all 2nd edition (and according to some other people’s experience, some 3rd ed) phones may reach a state where they are no longer to make TCP/IP connections, and the only cure is a reboot. We only use this in unattended applications (see http://loca-lab.org and http://article.no/exhibition/the-head/).

There are at three separate ways the phone may permanently fail to make TCP/IP connections:
- sometimes the name resolver gets an error when resolving a name, and will never retry the resolution
- sometimes the phone seems able to create a GPRS connection, but will never actually put any data through it
- sometimes the phone totally loses the ability to talk to the NOS/GSM side (unable to make calls, 0 signal strength).

When Nokia starts making phones that don’t require a reboot to work, we will stop rebooting them automatically.

Mika Raento
Forum Nokia Champion

Posted in Symbian | 14 Comments

Jaiku Mobile for S60 3rd ed. released!

For those of you living in a barrel, I’m posting here as well: the 3rd edition client for Jaiku has been officially released together with the new site! Go grab it from jaiku.com/.

We actually had a signed version a couple of weeks back (on the first try, no problems, using SwEvent, ReadDeviceData and WriteDeviceData), but found some bugs in early E61 firmware that had to be worked around, so it got postponed.

Posted in Symbian | Comments Off

Problems with 9.1 SDK tools

I’m just documenting what we have done here. These are not cookbook like. YMMV. Handle with care.

  • err_formatter.pl does not handle arguments in spaces in them correctly.

    The line that puts together the command must be changed to

    my $command = join(' ', map { '"' . $_ . '"' } @ARGV);

  • ide_cw.pm uses library names incompatible with command line builds, leading to KErrCorrupt on the device, creates resource XML files that use the same language for all generated resources, references non-existent DEF files for DLLs with EXPORTUNFROZEN, adds references to non-existent paths that generate warnings in CodeWarrior and uses optimization in debug builds with GCCE.

    See ide_cw_pm.diff for all the changes. It includes a couple of local hacks, and assumes that you either use DEF files and EXPORTUNFROZEN (not both), so be careful with it.

  • The XML files generated with abld makefile cw_ide don’t work for on-device debugging. You have to manually add the necessary options for SIS file creation each time you recreate the project. You can modify your cw_project_template_v4.xml to fix that.
  • If you want dependencies between projects in CodeWarrior, you have to manually add the sub projects each time you create the project. You can use something like cw.pl to automate the references. Again, it’s a bit hacky and you probably have to modify it to make it work with your projects.
Posted in Symbian | 3 Comments

3rd edition MMC writes take 50 times as long as writes to internal memory

We’ve had great problems running Jaiku from an MMC with acceptable performance. Symbian Signed requires you to only store data on the drive your application is installed to. Jaiku keeps a large number of small data items which are fairly frequently updated, stored in a Symbian database.

To get a better understanding of what the performance is actually like, I hacked the good old Bonnie disk benchmark to run on Symbian. Basic result: small writes to the MMC take 50 times as much time as to the internal memory.

In the table below, Block size is 1 KB. Rewrite actually writes only 1 byte per block but counts the throughput as full blocks. The absolute figures are likely to depend heavily on my implementation, but the relative performance should be quite reliable. The machines are Model-Disk, so N80-C means N80, internal memory, and N80-E means N80, MMC.

              -------Sequential Output-------- ---Sequential Input--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU

N80-C      16   173  0.0   285  0.0   590  0.0   968  0.0  2855  0.0
N80-E      16     3  0.0    10  0.0   163  0.0   551  0.0  1254  0.0

5500-C      4    75  0.0    50  0.0    44  0.0   780  0.0  1700  0.0
5500-E      4    11  0.0    16  0.0   133  0.0   391  0.0   747  0.0

N73-C      16   158  0.0   285  0.0   519  0.0   977  0.0  3048  0.0
N73-E      16     2  0.0     5  0.0   144  0.0   544  0.0  1274  0.0

E70-C      16   223  0.0   442  0.0   657  0.0  1004  0.0  3085  0.0
E70-E      16     5  0.0     5  0.0   161  0.0   552  0.0  1282  0.0

The exact details of the phones and memory cards are:

  • N80: V 4.0623.0.42, 20-09-2006, RM-92, Nokia N80 (03); Nokia 128MB MiniSD S128T
  • 5500: V 03.18, 07-09-06, RM-86, Nokia 5500 Sport; Nokia 64MB MicroSD C64T
  • N73: V.2.0628.0.0.1, 10-07-2006, RM-133, Nokia N73 (03); Nokia 64 MB MiniSD S64T
  • E70: V.1.0610.05.07, 30-05-06, RM-10 Nokia E70; Nokia 64MB MiniSD S64T
Posted in Symbian | 6 Comments

GCC-E and ARMCC compared

I finally got the ARM compiler evaluation CD and license today. Executive summary: code size down 30% (220 KB), and since that’s in compressed size my guess is that we are saving at least 400KB run-time memory consumption.

Code size with GCC-E, optimising for size

contextexpat.dll         size       41245
ContextNotify.exe        size       12709
contextnotifyclient.dll  size       14650
contextcommon.dll        size       73165
ContextServer.exe        size       22341
contextclient.dll        size        5418
blackboarddata.dll       size       38807
bbf200084BF.dll          size        3752
BlackBoardServer.exe     size       37302
BlackBoardClient.dll     size       19600
contextsensordata.dll    size       41278
contextsensors.dll       size       67359
bbf200084C2.dll          size        4476
contextcommsensors.dll   size        6313
ContextNetwork.dll       size       31886
ContextCommon2.dll       size       24204
contextUI.dll            size       25075
jaikuuikit.dll           size       19296
ContextContactsUi.dll    size      106666
jaikusettings.exe        size       44178
contextcontacts.exe      size       52026
cl_starter.exe           size        8616
contextwelcome.exe       size        6971
JaikuTool.exe            size        8428
total                    size      715761

Code size with ARMCC, optimising for size

contextexpat.dll         size       32210
ContextNotify.exe        size        7971
contextnotifyclient.dll  size        9503
contextcommon.dll        size       51096
ContextServer.exe        size       15153
contextclient.dll        size        2539
blackboarddata.dll       size       24688
bbf200084BF.dll          size        1293
BlackBoardServer.exe     size       26646
BlackBoardClient.dll     size       13678
contextsensordata.dll    size       21583
contextsensors.dll       size       46827
bbf200084C2.dll          size        1892
contextcommsensors.dll   size        2788
ContextNetwork.dll       size       23173
ContextCommon2.dll       size       16702
contextUI.dll            size       18663
jaikuuikit.dll           size       13256
ContextContactsUi.dll    size       73089
jaikusettings.exe        size       32944
contextcontacts.exe      size       38605
cl_starter.exe           size        6051
contextwelcome.exe       size        3922
JaikuTool.exe            size        5561
total                    size      489833

Contact ARM for pricing and purchasing. I think you get a better deal if you are a Symbian Platinum Partner and only need the compiler for Symbian.

Posted in Symbian | 5 Comments

Flaky, very flaky

Trying to crunch through all the robustness and stability testing necessary for Symbian signed I’m becoming sick and tired of the quality of the platform we are supposed to use. I have wasted countless hours with the following problems:

  • CodeWarrior crashes regularly when debugging. Several times a day, mostly with on-device debugging but now and then also on the emulator. I’ve sent the crash dumps to Nokia who we’ve paid quite a bit for the licenses, but nothing so far.
  • CodeWarrior keeps hidden references to projects, so after an MMP change I can’t reload the XML without restarting the whole IDE.
  • We have both Finnish and English resource files. Compiling the resources CodeWarrior generates the same output for both variants (Finnish). Resources have to be recompiled on the command line.
  • The N80 hangs with on-device-debugging _all the f*cking time_. I don’t know if the problem is with the App TRK, the USB implementation or the physical connection, but I can basically count on having to take out the battery after each debugging session (it does seem to be a problem with the physical connector on the phone).
  • The Symbian Signed criteria states that we should handle out-of-disk-space situations gracefully. Never mind that the platform doesn’t: App TRK sometimes hangs when filling disk space, and the application manager often fails to remove an application from a full disk without any explanation (no error display whatsoever, the uninstall just doesn’t happen). And Leaving with KErrDiskFull from a document’s ConstructL normally produces no user-visible feedback, you have to implement that yourself.
  • Even tho LOWMEM testing is waived for 9.1, we’ve been doing it on the emulator to find bugs in error handling and destructor code. Leaving from an AppUi’s ConstructL leaks memory, since the AppUi is not destructed by the framework in that case (and cannot safely be destructed manually).
  • The Bluetooth stack on the N80 seems to be almost as bad as back in the days of the 7650. Using it as a modem with my laptop or transferring files starts failing after a few times, and necessitates me to go to the phone and switch the bluetooth off and on to recover. Trying to transfer data from my older phone via bluetooth gives btnotifappserver -44. On an E70 you get the classic BTServer crashes.
  • To be able to step into code or set breakpoints in CodeWarrior, the project’s target has to be set to the active debugging target. This means that switching debugging between the emulator and the device requires you to switch the target in all your projects, for us in about 30. Remains to be scripted.
  • The scripts that generate CodeWarrior project import XML files are broken in multiple ways (some of which we’ve fixed by now, some that await fixing), for example: if you have no DEF file and specify EXPORTUNFROZEN, an imaginary base DEF file is still put to the linker command line, which then fails.
  • Trying to install a SIS file over bluetooth with PC Suite just gives ‘Operation failed’.
  • Giving the following backup_registration.xml to the phone makes the phone backup engine crash: a sbengine USER 130 panic. This would of course fail signing. Without the public_backup bit the backup engine doesn’t crash, but our files are not backed up because the application is not closed before the ‘Full backup’ stage, which is after public file backup.
    <?xml version="1.0" standalone="yes"?>
            <restore requires_reboot = "no"/>
  • An application running out of memory (exceeding maximum heap size) makes the phone reboot (yes, we’ve now both tuned the memory usage and upped the heap limit)
Posted in Symbian | 20 Comments

GCCE ‘support’ in the 3rd edition SDK

(update at 19:15, ho-hum, after the griping I found the Maintenance Release notes on Forum Nokia, which do recommend setting the optimization options in gcce.mk.)

It has been ‘generally known’ that binaries produced by GCCE (the CSL ARM gcc toolchain) for 9.1 are larger than what we’ve been used to. The Jaiku SIS file clocks in at about 700k on 2nd edition, but around 1400k on 3rd. The suggested solution is to purchase RVCT from ARM (for several thousand USD per seat).

I was digging into the compiler flags used because I was getting corrupt binaries (as in not being able to start on the device, giving error -20 (KErrCorrupt) instead) when building in the CodeWarrior IDE (another few thousand, but the only way to do on-device debugging).

Turns out that the Symbian/Nokia build tools enable no optimization whatsoever on GCCE. There’s just a comment in the ide_cw.pl:

#       gcc version 3.4.3 (release) (CodeSourcery ARM Q3cvs 2004) 
#       -- currently link fails when building with -O4
#       $compilerargs .= " -O4 "

I guess it goes without saying that the SDK is not using the Q3 2004 version, but a Q1 2005 one.

The SDK also states that ‘The Symbian OS builds tools apply the following policy when building projects: kernel-side code is built for ARM, while other code (user-side) is built for THUMB’. So does that mean your code is built with the THUMB instruction set? Of course not. It’s just in the SDK to confuse you.

So changing gcce.mk and ide_cw.pl to use -Os -mthumb drops the SIS file size down to 880k, which although not very nice compared to 670, is a whole lot better than 1400.

I’m now postponing purchasing RVCT and checking whether the compiler options break anything.

Maybe Nokia and Symbian could document if there are any known issues with optimization and the version of GCCE actually shipped. Even better, it would be nice to know whether we could upgrade to the 2005 Q3 version from CodeSourcery.

Ah, and the KErrCorrupt was caused by command-line tools thinking the link name of dlls to be something like jaikusettings{000a0000}[20006e42].exe and the IDE saying that it was ngs{000a0000}[0x20006e42].exe. Note the crucial 0x. Just another thing to modify in our version of the build scripts, together with specifying no optimization for the GCCE UDEB build when SRCDBG is given, just like with RVCT.

Posted in Symbian | 4 Comments