Periodic several second freezes with Raspberry Pi 3

[dankcushions]

I'm getting periodic (about once every few minutes, although it's not consistent) hangs of 2-5 seconds, even on games with very lowly/2d graphics like 'work time fun', and even when the game isn't doing anything.

The pi is running the roms from a USB drive, but my RAM isn't full and it doesn't happen during obvious loads, so i don't think it's necessarily a storage speed issue (unless there's other stuff going on!). the CPU is also down in the low %s when the freezes are happening.

I've rebuilt from the current version, disabled developer logging, but none of the other suggested performance options strike me as being likely culprits, so thought I'd ask here before changing everything from default and getting into a mess!

I temporarily enabled developer logging and got the attached log
(there were two freezes i noticed here about 5 seconds apart, and after the second i exited)

[dankcushions]

a little bump! i just retried this after building the latest version from git - still the same issue. interestingly, i've noticed it freezes the whole system - UI and all. as in, i can get the freezes whilst the UI was running whilst the game was paused?

i use the pi extensively with other emulators and don't get any freezes there, so i don't think it's a system problem, but i'm curious if anyone has a pi3 running this WITHOUT any freezes?

[arutema]

I have this on my Pi 3 as well. I'd been blaming my off-brand SD card, but if you're also having the same issue, it may indeed be a problem in PPSSPP.

Do you have a means like a SSH client to watch system stats while these happen? I was seeing iowait% spiking to 50 during these freezes.

You may also want to post your results in the "PPSSPP for Pi" thread in the development forum.

[arutema]

You've saved me a lot of time fiddling with multiple SD cards. And I present; a partial fix.

It appears PPSSPP does not get along well with deadline, the default i/o scheduler on Rasbian.

The first step is to change this to cfq. Using sudo, edit /boot/cmdline.txt. Change the text "elevator=deadline" to "elevator=cfq" without changing anything else. Reboot.

Then start PPSSPP, before loading a game, under Settings -> System, change "I/O timing method" to "Host".

You'll still get some hangs, but they will not be as frequent or as long as with the deadline scheduler.

[[Unknown]]

Deadline is a popular option among flash drives. CFQ is typically the default used with moving-parts disks.

If that affects it, it definitely indicates this is IO latency related. Are you using a cso or iso? How much memory does the device have? Could it be swapping - do you see swap usage?

Do pis come by default with zram configured, or do they use SD card swap, or no swap at all?

-[Unknown]

[arutema]

(06-03-2016 01:16 AM)[Unknown] Wrote:  Deadline is a popular option among flash drives. CFQ is typically the default used with moving-parts disks.

If that affects it, it definitely indicates this is IO latency related. Are you using a cso or iso? How much memory does the device have? Could it be swapping - do you see swap usage?

Do pis come by default with zram configured, or do they use SD card swap, or no swap at all?

-[Unknown]

Pi 3 comes with 1GB of ram, 128M of which has to be reserved for the GPU for PPSSPP, leaving about 860M for the OS and PPSSPP. A swapfile is on the SD card.

Looking at sar -r and sar -R, it looks like these hangs match up to drops in kbdirty, while swap usage stays at 0. I wonder if the default system settings are to flush writeback cache in big chunks to limit write-cycles to the SD card.

This wouldn't affect emulators for cartridge-based systems, since they typically keep the whole ROM image in memory.

I haven't tried PS1 emulation on the Pi 3 yet, but I suspect it may see similar issues. dankcushions, I don't suppose you have any reports from PS1 emulators?

sar excerpts attached. And for reference, all my games are uncompressed ISO or PSN store downloads.

[dankcushions]

thanks very much for the replies!

(06-03-2016 01:16 AM)[Unknown] Wrote:  Deadline is a popular option among flash drives. CFQ is typically the default used with moving-parts disks.

If that affects it, it definitely indicates this is IO latency related. Are you using a cso or iso?

ISO for my work time fun example.

Quote:How much memory does the device have? Could it be swapping - do you see swap usage?

Do pis come by default with zram configured, or do they use SD card swap, or no swap at all?

-[Unknown]

1GB of RAM (256MB allocated to GPU in my case). it's not swapping. in fact, PPSSPP doesn't seem to take a whole bunch of memory up (shouldn't it be loading this whole ISO into memory? WTF is 439MB).

here's the vmstat -a output during a freeze:

Code:

0  1      0 343180 164580 203588    0    0     0   396 2502 2662  1  1 75 23  0
0  0      0 343180 164580 203588    0    0     0     0 2741 3772  5  1 91  4  0
0  0      0 343244 164580 203588    0    0     0     0 2753 3834  6  1 93  0  0
0  0      0 343244 164580 203588    0    0     0     0 2756 3772  6  5 89  0  0
^C
pi@retropie:~ $ vmstat -a 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa st
0  1      0 342996 164580 203492    0    0     0     1    7   13  6  1 93  0  0
0  2      0 342996 164580 203596    0    0     0   324 2624 3283  3  0 60 37  0
0  1      0 342996 164580 203596    0    0     0     4 2778 3904  6  2 73 19  0

the freezes occured when bo spiked (I/O blocks sent). does this mean it's sending data to the SD card? why would it be doing this? no swap, and the game isn't loading anything. weird!

(06-03-2016 03:45 AM)arutema Wrote:  I haven't tried PS1 emulation on the Pi 3 yet, but I suspect it may see similar issues. dankcushions, I don't suppose you have any reports from PS1 emulators?

pcsx_rearmed is flawless on the pi3 (and pi2, pretty much).

[[Unknown]]

PPSSPP doesn't load the entire ISO into RAM; in general, it relies on your operating system to do this if it can. For 64-bit versions we do have an option to load the entire (potentially 1.8GB) ISO into RAM.

You might try using CSOs. At least, this will reduce IO. The gunzip overhead is not that huge.

Hmm. I wonder if it's writing to log files or something? The main ways PPSSPP will write are:

1. Save data, but this shouldn't be frequent.
2. Save states and screenshots, but only when you trigger them.
3. INI file updates, only when you change settings.
4. Log file (which will be nearly constant) if enabled.
5. Textures for replacement, if SaveNewTextures is enabled (off by default.)
6. HTTP disk cache, if you're using an http:// URL.

You could try starting with --log=/dev/null, I suppose, but I doubt that will help since you're not logging to a file, anyway.

You can check /proc/(pid of PPSSPP)/fd. If you ls -lh it, you'll see the file handles it has open. You could also verify it's PPSSPP with iotop.

Oh - I just had an idea. We memory map an anonymous file to represent PSP RAM. This shouldn't be written to disk at all, but maybe it is. That would write quite often, if it was getting flushed to disk for some reason.

https://github.com/hrydgard/ppsspp/blob/...a.cpp#L184

-[Unknown]

[arutema]

(06-03-2016 01:27 PM)[Unknown] Wrote:  Oh - I just had an idea. We memory map an anonymous file to represent PSP RAM. This shouldn't be written to disk at all, but maybe it is. That would write quite often, if it was getting flushed to disk for some reason.

https://github.com/hrydgard/ppsspp/blob/...a.cpp#L184

-[Unknown]

This looks to be the issue. I had PPSSPP save this to a tmpfs and everything runs smooth now.

I changed the block at https://github.com/hrydgard/ppsspp/blob/...na.cpp#L93 from:

Code:

#ifndef _WIN32
// do not make this "static"
#ifdef MAEMO
std::string ram_temp_file = "/home/user/gc_mem.tmp";
#elif defined(BB)
std::string ram_temp_file = "/home/root/gc_mem.tmp";
#else
std::string ram_temp_file = "/tmp/gc_mem.tmp";
#endif
#elif !defined(_XBOX)
SYSTEM_INFO sysInfo;
#endif

to

Code:

#ifndef _WIN32
// do not make this "static"
#ifdef MAEMO
std::string ram_temp_file = "/home/user/gc_mem.tmp";
#elif defined(BB)
std::string ram_temp_file = "/home/root/gc_mem.tmp";
#elif defined(RPI)
std::string ram_temp_file = "/dev/shm/gc_mem.tmp";
#else
std::string ram_temp_file = "/tmp/gc_mem.tmp";
#endif
#elif !defined(_XBOX)
SYSTEM_INFO sysInfo;
#endif

I'm not quite sure if this should be based on RPI or RASPBIAN being defined. I don't feel like playing around with other distros tonight.

[[Unknown]]

Hmm, well, that's both good to know, and concerning. This is basically just Debian, right? I'm now worried this is happening on other Linux systems. We use an entirely different mechanism on Android (ashmem.)

Probably we should check for /dev/shm, and if it exists, use it (on any Unix variant.) I know some will mount /tmp as tmpfs by default.

-[Unknown]

[[Unknown]]

See here for an implementation of that:

https://github.com/hrydgard/ppsspp/pull/8801

-[Unknown]

[dankcushions]

thanks so much for this, al!l you've just made PSP emulation on a whole bunch of SOCs viable! so awesome!