How to access physical addresses from user space in Linux?
On a ARM based system running Linux, I have a device that’s memory mapped to a physical address. From a user space program where all addresses are virtual, how can I read content from this address?
2 Answers 2
busybox devmem
busybox devmem is a tiny CLI utility that mmaps /dev/mem .
You can get it in Ubuntu with: sudo apt-get install busybox
Usage: read 4 bytes from the physical address 0x12345678 :
sudo busybox devmem 0x12345678 Write 0x9abcdef0 to that address:
sudo busybox devmem 0x12345678 w 0x9abcdef0 mmap MAP_SHARED
When mmapping /dev/mem , you likely want to use:
open("/dev/mem", O_RDWR | O_SYNC); mmap(. PROT_READ | PROT_WRITE, MAP_SHARED, . ) MAP_SHARED makes writes go to physical memory immediately, which makes it easier to observe, and makes more sense for hardware register writes.
CONFIG_STRICT_DEVMEM and nopat
To use /dev/mem to view and modify regular RAM on kernel v4.9, you must fist:
- disable CONFIG_STRICT_DEVMEM (set by default on Ubuntu 17.04)
- pass the nopat kernel command line option for x86
IO ports still work without those.
Cache flushing
If you try to write to RAM instead of a register, the memory may be cached by the CPU: How to flush the CPU cache for a region of address space in Linux? and I don’t see a very portable / easy way to flush it or mark the region as uncacheable:
So maybe /dev/mem can’t be used reliably to pass memory buffers to devices?
This can’t be observed in QEMU unfortunately, since QEMU does not simulate caches.
How to test it out
Now for the fun part. Here are a few cool setups:
- Userland memory
- allocate volatile variable on an userland process
- get the physical address with /proc//maps + /proc//pagemap
- modify the value at the physical address with devmem , and watch the userland process react
- allocate kernel memory with kmalloc
- get the physical address with virt_to_phys and pass it back to userland
- modify the physical address with devmem
- query the value from the kernel module
- create a platform device with known physical register addresses
- use devmem to write to the register
- watch printf s come out of the virtual device in response
Bonus: determine the physical address for a virtual address
Difference between physical/logical/virtual memory address
My answer is true for Intel CPUs running on a modern Linux system, and I am speaking about user-level processes, not kernel code. Still, I think it’ll give you some insight enough to think about the other possibilities
Address Types
I have come across discussion that virtual and logical addresses/address space are the same. Is it true?
As far as I know they are the same, at least in modern OS’s running on top of Intel processors.
Let me try to define two notions before I explain more:
- Physical Address: The address of where something is physically located in the RAM chip.
- Logical/Virtual Address: The address that your program uses to reach its things. It’s typically converted to a physical address later by a hardware chip (mostly, not even the CPU is aware really of this conversion).
Virtual/Logical Address
The virtual address is well, a virtual address, the OS along with a hardware circuit called the MMU (Memory Management Unit) delude your program that it’s running alone in the system, it’s got the whole address space(having 32-bits system means your program will think it has 4 GBs of RAM; roughly speaking).
Obviously, if you have more than one program running at the time (you always do, GUI, Init process, Shell, clock app, calendar, whatever), this won’t work.
What will happen is that the OS will put most of your program memory in the hard disk, the parts it uses the most will be present in the RAM, but hey, that doesn’t mean they’ll have the address you and your program know.
Example: Your process might have a variable named (counter) that’s given the virtual address 0xff (imaginably. ) and another variable named (oftenNotUsed) that’s given the virtual address (0xaa).
If you read the assembly of your compiled code after all linking’s happened, you’ll be accessing them using those addresses but well, the (oftenNotUsed) variable won’t be really there in RAM at 0xaa, it’ll be in the hard disk because the process is not using it.
Moreover, the variable (counter) probably won’t be physically at (0xff), it’ll be somewhere else in RAM, when your CPU tries to fetch what’s in 0xff, the MMU and a part of the OS, will do a mapping and get that variable from where it’s really available in the RAM, the CPU won’t even notice it wasn’t in 0xff.
Now what happens if your program asks for the (oftenNotUsed) variable? The MMU+OS will notice this ‘miss’ and will fetch it for the CPU from the Harddisk into RAM then hand it over to the CPU as if it were in the address (0xaa); this fetching means some data that was present in RAM will be sent back to the Harddisk.
Now imagine this running for every process in your system. Every process thinks they have 4GB of RAMs, no one actually have that but everything works because everyone has some parts of their program available physically in the RAM but most of the program resides in the HardDisk. Don’t confuse this part of the program memory being put in HD with the program data you can access through file operations.
Summary
Virtual address: The address you use in your programs, the address that your CPU use to fetch data, is not real and gets translated via MMU to some physical address; everyone has one and its size depends on your system(Linux running 32-bit has 4GB address space)
Physical address: The address you’ll never reach if you’re running on top of an OS. It’s where your data, regardless of its virtual address, resides in RAM. This will change if your data is sent back and forth to the hard disk to accommodate more space for other processes.
All of what I have mentioned above, although it’s a simplified version of the whole concept, is what’s called the memory management part of the the computer system.
Consequences of this system
- Processes cannot access each other memory, everyone has their separate virtual addresses and every process gets a different translation to different areas even though sometimes you may look and find that two processes try to access the same virtual address.
- This system works well as a caching system, you typically don’t use the whole 4GB you have available, so why waste that? let others share it and let them use it too; when your process needs more, the OS will fetch your data from the HD and replace other process’ data, at an expense of course.