Wednesday, December 2, 2015

Linux: Compare if two files are the same

Quick script I wrote in my .bashrc to check if two files are the same:

#md5sum
function md5comp()
{
    a=$(md5sum $1 | awk '{print $1}')
    b=$(md5sum $2 | awk '{print $1}')
    echo "Comparing $1 and $2"
    [ "$a" = "$b" ] && echo  "Equal" || echo "Not Equal"
}

Usage:
$ md5comp filea.txt fileb.txt
Comparing filea.txt and fileb.txt
Equal

Wednesday, November 18, 2015

Linux: Why to drop caches and how? Host or virtual machine

Excerpt from stackoverflow: "The reason to drop caches like this is for benchmarking disk performance, and is the only reason it exists. When running an I/O-intensive benchmark, you want to be sure that the various settings you try are all actually doing disk I/O, so Linux allows you to drop caches rather than do a full reboot."

How to drop caches on the host?

Monday, September 28, 2015

Google Blogger Header Image Design

Do you like our new header image on the softlockup.com website? If you do, you can make one for your blog too. We used a nice open source software called inkscape.

If you want to learn the ropes with inkscape, there are multiple tutorials on youtube, which are easily obtained on searching. 

I will put a gist of the steps we used to create the image you see below:
Our blog's header image

  1. Create or obtain a logo: Either use a professional service or use your ideas to create a logo for your website. This part is optional. You can create the header image without a logo although it looks better with one.

Thursday, September 24, 2015

Why Linux Kernel KASLR is not very effective

Recently, with more time on hand  I am reading about security in the Linux kernel. A common mode of attack on any program is using buffer overflow to implement return oriented programming (ROP) blobs. Return oriented programming is a mechanism of overwriting return addresses in a library to implement code blobs (or gadgets) that will perform the desired functionality.

Saturday, September 5, 2015

Which Linux Kernel timing or delay APIs to use for what?

I classify the waiting or timing API in the Linux kernel in two categories:
1. Which blocks the current thread of execution.
2. Something which has to be scheduled for later but we want the current thread to continue.
In most cases, the distinction between 1 and 2 is clear, but the techniques used to implement 2 can also be manipulated to behave like 1.



1. Blocking current thread of execution (Inline delays)
The API for 1. in the above case are:

Tuesday, June 23, 2015

p4 shelve equivalent in Git with an example

The equivalent of p4 shelve on GIT is stashing.

Say I have modified the file on my repository and I see the following:
techmuser@gw2:~/repositories/softlockup_repo/scripts$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)
        modified:   vimrc
no changes added to commit (use "git add" and/or "git commit -a")
Now to shelve these changes, 
techmuser@gw2:~/repositories/softlockup_repo/scripts$ git stash
Saved working directory and index state WIP on master: 80e892a Track git aliases.       modified:   generic_aliases
HEAD is now at 80e892a Track git aliases.

Monday, June 22, 2015

Implementing tftpboot with Linux u-boot

If your u-boot is not running tftpboot, for any reason as is the case with my board, you can always fool it into doing tftpboot for you by leveraging the run support in the u-boot bootloader.

U-boot is a very popular bootloader and you can learn more about it here: http://www.denx.de/wiki/U-Boot

Here is the trick to implement it:
setenv bootcmd 'dhcp; run cmd'
setenv cmd 'tftp LOADADDR PATH_TO_TFTPIMAGE'

 In the above set of commands you will have to replace LOADADDR with the correct address in memory where you want to load the image. The PATH_TO_TFTPIMAGE is the path to the image on the tftpserver.

Saturday, June 20, 2015

Why not have multiple woofers for each channel

Ever wondered why there is a single subwoofer and multiple small speakers for higher frequencies?
Because the human hear can only distinguish the higher frequencies spatially. Why?
Because of the wavelength. The base (subwoofer) is the lower frequency sound which is usually a few HZ which would typically have a wavelength of the order of 10s of feet. Hence in this case, the two human ears which are placed less than a feet apart on our head cant differentiate between base sound  from different channels. Hence we only have one subwoofer for the base sounds and multiple smaller  speakers (5.1 or 2.1 stereo) for the higher frequency sounds.

Monday, April 6, 2015

Creating a disk catalog for your external hard drive - Windows

On my windows machine, I wanted to create a disk catalog that I could use to see the movies I already had watched or downloaded from my provider. 
  1. I found that the easiest way to do this is by creating a text file which has a list of all files on the external drive from that directory and saving that info to a file.
  2. When needed, search the text file which is used as a catalog.

Creation of the catalog is very easy. For your destination path (on the external drive),  do the following:
  1. From the windows start menu, type and search for "cmd"
  2. This will launch the command window.
  3. In the command window run the following command:
DIR "F:\path\to\dir" /s > "C:\List.txt"
This command assumes that F:\path\to\dir is the directory which you are planning to list. The catalog file is the C:\List.txt file which can be read and searched like a normal text file.

Monday, March 9, 2015

Organizing aliases on Linux

Just because I wanted to share and track the aliases I use on my linux box across multiple boxes, including VMs, I decided to put the .aliases file on github.

However, my .aliases was littered with tidbits and shortcuts that I use specifically for work which wont be relevant across my other machines. To remedy this problem I decide to do the following:

  1. Split my original .aliases file into two files: work_aliases and generic_aliases.
  2. From my .bashrc I now invoke two separate aliases as
source ~/generic_aliases
source~/work_aliases 

Now I can conevniently manage and track the aliases and shortcuts. I have shared my collection of useful shortcuts here:
https://github.com/techmuser/softlockup_repo/blob/master/scripts/generic_aliases

As always:
Complete source can be downloaded here
Complete repo of free source code can be downloaded here 

Tuesday, March 3, 2015

How to check the IO address space on your Linux machine

Parallel port is at address 0x0378?
This is the IO address space and this information is provided under the /proc file system as:

$> cat /proc/ioports

Sample output is as follows:

Monday, March 2, 2015

Explanation and code for reading interface names and MAC addresses

Here is a useful piece of user space code that can read the interfaces and their corresponding MAC addresses.

Download complete source here.
Complete repository can be downloaded or cloned here.

Main parts of the code:

The main parts of the program are:

1. Given the name of the interface, execute an IOCTL to get the hardware address:
/* gets the MAC addr for the interface */
int get_if_mac(char *eth, char *mac) {
    struct ifreq ifr;
    int fd = socket(AF_INET, SOCK_DGRAM, 0);
    if (fd == -1) return -1;
    strncpy(ifr.ifr_name, eth, sizeof(ifr.ifr_name));
    if (ioctl(fd, SIOCGIFHWADDR, &ifr) == -1) return -1;
    memcpy(mac, ifr.ifr_hwaddr.sa_data, IFHWADDRLEN);
    return 0;
}

2. Get interface info from the system in a linkedlist ifaddr:
 if (getifaddrs(&ifaddr) == -1) {
        perror("getifaddrs");
        exit(EXIT_FAILURE);
    }
3.  Loop and call the function get_if_mac() in step 1.

Wednesday, February 11, 2015

One line difference between delivery enabled and trigger enabled in U-APSD

Excerpt from wikipedia:
Queues may be configured to be trigger enabled, (i.e. a receipt of a data frame corresponding to the queue acts as trigger), and delivery enabled, (i.e. data stored at those queues will be released upon receipt of a frame). Queues refer to the four ACs defined for WMM.

Thursday, February 5, 2015

Brief differences between tasklets and Softirqs

Formulated this comparison based on books and posts I read so far:

Click on the image to enlarge.

Explanation and code snippet to read hotplug events from a netlink socket

Complete source file can be downloaded here:
Source code for this program is here.
Complete repository can be downloaded or cloned here.

Parts of the code are as explained below:
The kernel notifies the userspace like (udev) about new available devices by  (1) Spawning a process or (2) a packet to a netlink socket on attach. Code taken directly from the kernel documentation:
  #include <stdio.h>
  #include <stdlib.h>
  #include <string.h>

  #include <sys/poll.h>
  #include <sys/socket.h>
  #include <sys/types.h>
  #include <unistd.h>

  #include <linux/types.h>
  #include <linux/netlink.h>
Error handling:
  void die(char *s)
  {
 write(2,s,strlen(s));
 exit(1);
  }
Main file:
  int main(int argc, char *argv[])
  {
 struct sockaddr_nl nls;
 struct pollfd pfd;
 char buf[512];

 // Open hotplug event netlink socket

 memset(&nls,0,sizeof(struct sockaddr_nl));
 nls.nl_family = AF_NETLINK;
 nls.nl_pid = getpid();
 nls.nl_groups = -1;

 pfd.events = POLLIN;
 pfd.fd = socket(PF_NETLINK, SOCK_DGRAM, NETLINK_KOBJECT_UEVENT);
 if (pfd.fd==-1)
  die("Not root\n");
Listen to the socket:
 if (bind(pfd.fd, (void *)&nls, sizeof(struct sockaddr_nl)))
  die("Bind failed\n");
 while (-1!=poll(&pfd, 1, -1)) {
  int i, len = recv(pfd.fd, buf, sizeof(buf), MSG_DONTWAIT);
  if (len == -1) die("recv\n");
Print the data once it is read:
  i = 0;
  while (i<len) {
   printf("%s\n", buf+i);
   i += strlen(buf+i)+1;
  }
 }
 die("poll\n");

 // Dear gcc: shut up.
 return 0;
  }
Running the code:

Wednesday, January 21, 2015

Solution: Fatal section header offset is bigger than file size

The error I was seeing while recompiling a driver:
fatal section header offset 32425246532452 in file 'vmlinux' is bigger than filesize=35524847

What helped was trying from scratch:
sudo make distclean
make menuconfig
make modules
sudo make modules_install

Wednesday, January 14, 2015

Perforce: Protected namespace access denied

I ran into this error when I was creating a new clientspec and tried to checkout a new tree with this clientspec. I was using
p4 sync ...
Protected namespace ... access denied

Found out through debugging that this error was not being generated due to a problem with the permissions on the local directory but rather because the clientspec was incorrectly pointing to a non-existing path on the repository. You are trying to access some part of the repo that is not setup or you dont have permissions to access. Run p4 client and fix the clientspec, and things should work.

Fix the client spec and things work.

Friday, January 9, 2015

Why do we need an executable stack with nested functions in GCC

  • The stack needs to be writable because the trampoline code is written on the stack which needs to be executed (to jump to the nested function). 
A nested function has no linkage by itself. So the trampoline code actually ensures two things:
  1. The outer () functions stack frame is available in the nested function
  2. Jump and execute code from the nested function.
Some other notes:
  • Nested functions are not a part of ANSI C, however, they are part of Gnu C.
  • The only reason nested functions are possibly useful because they use common stack variables.

Wednesday, January 7, 2015

Apple IOS: How to backup health data to windows or MAC

Before starting you need to have email correctly configured on your phone.
  1. Start the health app
  2. Navigate through "Health Data (folder with heart on it)" -> "All" -> Share icon in top right corner (the square box with the arrow on it).
  3. When you do that you will be prompted for confirmation as below:
    Health app prompting for confirmation before backing up
  4. Check your email. You should have the exported data in XML format.
Making sense of the XML data:
I found a note on an apple forum that neatly summarizes how you can view this information:
1) Open a blank Excel Workbook.
2) For Excel 2007 or more recent, choose the Data tab on the top.
3) On the left side there is a 'Get External Data' block of choices, click the dropdown arrow on 'From Other Sources'.
4) Click on the third choice 'From XML Data Import'.
5) Find your downloaded XML file and click 'Open'.
6) Excel will likely display a message that the source does not refer to a schema. Choose 'OK'.
7) If you click 'OK' with the default choice 'XML table in existing worksheet', data will be displayed in Excel table format, which will let you sort and filter data. The other choices only displays the data in columns.

You will likely need to create an additional column for dates, as the 'startDate' and 'endDate' times are in a format that Excel won't recognize, if you need any date to your data anyway. If you need a conversion formula you can reply to this message stating as much. You probably already found that the export zip file shows detailed logs for every day on an hourly basis for the last week only, prior records at least in the export are summarized by day.

Known issues:
I do not have a solution for any of these and I ran into them when I was backing up from my phone. So if you find something please comment.
  • Health app crashes (on startup)
  • Health app crashes while creating a backup.
  • Export is unsuccessful with the following screen seen below (could not export health data):
    Health app fails saying could not export data

Tuesday, January 6, 2015

How to Apply a Patch to the Linux Kernel Stable Tree

A nice small tutorial of how patching is done to the linux kernel tree by the LinuxFoundation.


Monday, January 5, 2015

Why would you always choose ext3 over ext2 filesystem?

The answer in one word would be: JOURNALING.


ext2 does not support journaling while ext3 does. So does ext4 (if you are ok with an in-the-works kind of a filesystem).

What does journaling do?
Excerpt from the ubuntu help page:
A journaling file system is more reliable when it comes to data storage. Journaling file systems do not necessarily prevent corruption, but they do prevent inconsistency and are much faster at file system checks than non-journaled file systems. If a power failure happens while you are saving a file, the save will not complete and you end up with corrupted data and an inconsistent file system. Instead of actually writing directly to the part of the disk where the file is stored, a journaling file system first writes it to another part of the hard drive and notes the necessary changes to a log, then in the background it goes through each entry to the journal and begins to complete the task, and when the task is complete, it checks it off on the list. Thus the file system is always in a consistent state (the file got saved, the journal reports it as not completely saved, or the journal is inconsistent (but can be rebuilt from the file system)). Some journaling file systems can prevent corruption as well by writing data twice.
A more elaborate comparison of the filesystems from the same page is as below:

File System
Max File Size
Max Partition Size
Journaling
Notes
Fat16
2 GB
2 GB
No
Legacy
Fat32
4 GB
8 TB
No
Legacy
NTFS
2 TB
256 TB
Yes
(For Windows Compatibility) NTFS-3g is installed by default in Ubuntu, allowing Read/Write support
ext2
2 TB
32 TB
No
Legacy
ext3
2 TB
32 TB
Yes
Standard linux filesystem for many years. Best choice for super-standard installation.
ext4
16 TB
1 EB
Yes
Modern iteration of ext3. Best choice for new installations where super-standard isn't necessary.
reiserFS
8 TB
16 TB
Yes
No longer well-maintained.
JFS
4PB
32PB
Yes (metadata)
Created by IBM - Not well maintained.
XFS
8 EB
8 EB
Yes (metadata)
Created by SGI. Best choice for a mix of stability and advanced journaling.
GB = Gigabyte (1024 MB) :: TB = Terabyte (1024 GB) :: PB = Petabyte (1024 TB) :: EB = Exabyte (1024 PB)
Above you'll see a brief comparison of two main attributes of different filesystems, the max file size and the largest a partition of that data can be.
Of the above file systems the only one you cannot install Linux on is the NTFS. It is not recommended to install Linux on any type of FAT file system, because FAT does not have any of the permissions of a true Unix FS.

Friday, January 2, 2015

Linux kernel makefile system in brief

There are multiple components in the kernel build system as shown in the table below:


Components in the Linux kernel build system

The Top makefile builds kernel image and modules. It does this by visiting directories recursively based on the kernel .config. The top Makefile textually includes an arch Makefile with the name arch/$(ARCH)/Makefile. The arch Makefile supplies architecture-specific information to the top Makefile. Each directory has a kbuild makefile which builds built-in or modular targets based on commands passed from above and info n the .config. Makefiles in the kernel are also referred to as kbuild files.

Further reading: A detailed discussion on these is available in the kernel documentation.