Btrfs

2014-08-09
, , , , , , , , , , ,
I'm using this logo .. because reasons
I'm using this logo .. because reasons

(on Wikipedia)
https://btrfs.wiki.kernel.org/

A filesystem I use because of checksumming and deduplication.

Very good, although I've had a sketchy history with it. I've used multiple versions of it, sometimes mounting the same partition with different versions of it.



I've heard that Btrfs does not support swap files.

--

  • Some time thereafter I stopped using btrfs to simplify my life. The problems I've had were major disasters requiring a huge learning curve, and that's intolerable. I'd rather waste drive space in having no deduplication than deal with btrfs barfing again.
  • 2016-03-28 - 3.18.2, on Slackware 14.1

  • 2016-03-23 - 3.12, on Lubuntu 14.04.4 LTS

Table of Contents [hide]

2016-03-28 - 3.18.2 ∞

This comes with Slackware 14.1

2016-03-23 - 3.12 ∞

3.12-1ubuntu0.1

This comes with Lubuntu

I've been using this for some time, and I've been happy with it.

Some time recently, I had some more silent data corruption. It could be this, or it could be eCryptfs.

btrfs fucked up again ∞

This is the third nightmare I've had with this filesystem. I'm sick of it.

How it started ∞

I think this all started with my attempting to use the top HDD dock in my case.

I rebooted harshly in the middle of moving some files around, and now btrfs is semi-fucked. I'm going to migrate away from it once I figure out what I'd need to restore from this nuisance.

I could read, but copying data is slow.. and copying a tiny bit of data then attempting to sync won't do anything. No i/o activity is displayed with htop.

One of my cores will mysteriously spike to 100% even though no process is shown to be using it. Sigh.

btrfs scrub ∞

So I tried a btrfs scrub. At first I was getting:

ERROR: getting dev info for scrub failed: Inappropriate ioctl for device

.. updating my btrfs tools resolved this issue.

I did a btrfs scrub and it did what it could but has unrecoverable errors.

btrfsck, smartctl, dmesg ∞

I ran btrfsck and it showed lots of errors. I did not run it with --repair

smartctl says the drive is fine, but it does show suspicious heat issues spiking to 58 C. It's bloody hot to touch. I cooled it down and re-tested, but there was no change.

I was also worried about physical damage since I did recently drop this drive. Yay me. Since other partitions were working well, and smart wasn't complaining, my worries were alleviated.

dmesg

[ 2192.531213] sd 2:0:0:0: [sdb] 7814037168 512-byte logical blocks: (4.00 TB/3.63 TiB)
[ 2192.531225] sd 2:0:0:0: Attached scsi generic sg1 type 0
[ 2192.531469] sd 2:0:0:0: [sdb] Write Protect is off
[ 2192.531483] sd 2:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[ 2192.531567] sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 2192.568545]  sdb: sdb1 sdb2 sdb3 sdb4
[ 2192.569555] sd 2:0:0:0: [sdb] Attached SCSI disk
[ 2193.025759] btrfs: device label data2 devid 1 transid 69015 /dev/sdb4
[ 2193.033683] btrfs: device label data2 devid 1 transid 69015 /dev/sdb4
[ 2193.051668] btrfs: device label data1 devid 1 transid 25471 /dev/sdb3
[ 2193.058922] btrfs: device label data1 devid 1 transid 25471 /dev/sdb3
[ 2199.894493] btrfs: device label data2 devid 1 transid 69015 /dev/sdb4
[ 2199.895882] btrfs: disk space caching is enabled
[ 2199.996917] btrfs: bdev /dev/sdb4 errs: wr 0, rd 2, flush 0, corrupt 831, gen 0

btrfsck ∞

\btrfsck <drive> --repair
btrfsck

enabling repair mode
Checking filesystem on /dev/sdb4
UUID: 84d22b45-b563-40ee-b7c7-be19887b74db
checking extents
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
root 5 inode 11181947 errors 1000, some csum missing
root 5 inode 11824946 errors 1000, some csum missing
root 5 inode 11824947 errors 1000, some csum missing
root 5 inode 11824948 errors 1000, some csum missing
root 5 inode 11824949 errors 1000, some csum missing
root 5 inode 11824950 errors 1000, some csum missing
root 5 inode 11824951 errors 1000, some csum missing
root 5 inode 11824952 errors 1000, some csum missing
root 5 inode 11824953 errors 1000, some csum missing
root 5 inode 11824954 errors 1000, some csum missing
root 5 inode 11824959 errors 1000, some csum missing
root 5 inode 11824960 errors 1000, some csum missing
root 5 inode 11824961 errors 1000, some csum missing
root 5 inode 11824962 errors 1000, some csum missing
root 5 inode 11824963 errors 1000, some csum missing
root 5 inode 11824964 errors 1000, some csum missing
root 5 inode 11824965 errors 1000, some csum missing
root 5 inode 11824966 errors 1000, some csum missing
root 5 inode 11824967 errors 1000, some csum missing
root 5 inode 11824968 errors 1000, some csum missing
root 5 inode 11824969 errors 1000, some csum missing
root 5 inode 11824970 errors 1000, some csum missing
root 5 inode 11824971 errors 1000, some csum missing
root 5 inode 11824972 errors 1000, some csum missing
root 5 inode 11824973 errors 1000, some csum missing
root 5 inode 11824974 errors 1000, some csum missing
root 5 inode 11824975 errors 1000, some csum missing
root 5 inode 11824976 errors 1000, some csum missing
root 5 inode 11824977 errors 1000, some csum missing
root 5 inode 11824978 errors 1000, some csum missing
root 5 inode 11824979 errors 1000, some csum missing
root 5 inode 11824980 errors 1000, some csum missing
root 5 inode 11824983 errors 1000, some csum missing
root 5 inode 11824984 errors 1000, some csum missing
root 5 inode 11824985 errors 1000, some csum missing
root 5 inode 11824986 errors 1000, some csum missing
root 5 inode 11824987 errors 1000, some csum missing
root 5 inode 11824988 errors 1000, some csum missing
root 5 inode 11824989 errors 1000, some csum missing
root 5 inode 11824990 errors 1000, some csum missing
root 5 inode 11824991 errors 1000, some csum missing
root 5 inode 11824992 errors 1000, some csum missing
root 5 inode 11824993 errors 1000, some csum missing
root 5 inode 11824994 errors 1000, some csum missing
root 5 inode 11824995 errors 1000, some csum missing
root 5 inode 11824996 errors 1000, some csum missing
root 5 inode 11824997 errors 1000, some csum missing
root 5 inode 11824998 errors 1000, some csum missing
root 5 inode 11824999 errors 1000, some csum missing
root 5 inode 11825000 errors 1000, some csum missing
root 5 inode 11825001 errors 1000, some csum missing
root 5 inode 11825002 errors 1000, some csum missing
root 5 inode 11825005 errors 1000, some csum missing
root 5 inode 11825006 errors 1000, some csum missing
root 5 inode 11825007 errors 1000, some csum missing
root 5 inode 11825008 errors 1000, some csum missing
root 5 inode 11825009 errors 1000, some csum missing
root 5 inode 11825010 errors 1000, some csum missing
root 5 inode 11825011 errors 1000, some csum missing
root 5 inode 11825012 errors 1000, some csum missing
root 5 inode 11825013 errors 1000, some csum missing
root 5 inode 11825014 errors 1000, some csum missing
root 5 inode 11825015 errors 1000, some csum missing
root 5 inode 11825016 errors 1000, some csum missing
root 5 inode 11825017 errors 1000, some csum missing
root 5 inode 11825018 errors 1000, some csum missing
root 5 inode 11825019 errors 1000, some csum missing
root 5 inode 11825020 errors 1000, some csum missing
root 5 inode 11825021 errors 1000, some csum missing
root 5 inode 11825022 errors 1000, some csum missing
root 5 inode 11825023 errors 1000, some csum missing
root 5 inode 11825026 errors 1000, some csum missing
root 5 inode 11825027 errors 1000, some csum missing
root 5 inode 11825028 errors 1000, some csum missing
root 5 inode 11825029 errors 1000, some csum missing
root 5 inode 11825030 errors 1000, some csum missing
root 5 inode 11825031 errors 1000, some csum missing
root 5 inode 11825032 errors 1000, some csum missing
root 5 inode 11825033 errors 1000, some csum missing
root 5 inode 11825034 errors 1000, some csum missing
root 5 inode 11825035 errors 1000, some csum missing
root 5 inode 11825036 errors 1000, some csum missing
root 5 inode 11825037 errors 1000, some csum missing
root 5 inode 11825038 errors 1000, some csum missing
root 5 inode 11825039 errors 1000, some csum missing
root 5 inode 11825043 errors 1000, some csum missing
root 5 inode 11825044 errors 1000, some csum missing
root 5 inode 11825045 errors 1000, some csum missing
root 5 inode 11825046 errors 1000, some csum missing
root 5 inode 11825047 errors 1000, some csum missing
root 5 inode 11825048 errors 1000, some csum missing
root 5 inode 11825049 errors 1000, some csum missing
root 5 inode 11825050 errors 1000, some csum missing
root 5 inode 11825051 errors 1000, some csum missing
root 5 inode 11825052 errors 1000, some csum missing
root 5 inode 11825053 errors 1000, some csum missing
root 5 inode 11825054 errors 1000, some csum missing
root 5 inode 11825055 errors 1000, some csum missing
root 5 inode 11825057 errors 1000, some csum missing
root 5 inode 11825058 errors 1000, some csum missing
root 5 inode 11825059 errors 1000, some csum missing
root 5 inode 11825060 errors 1000, some csum missing
root 5 inode 11825061 errors 1000, some csum missing
root 5 inode 11825062 errors 1000, some csum missing
root 5 inode 11825063 errors 1000, some csum missing
root 5 inode 11825064 errors 1000, some csum missing
root 5 inode 11825065 errors 1000, some csum missing
root 5 inode 11825066 errors 1000, some csum missing
root 5 inode 11825067 errors 1000, some csum missing
root 5 inode 11825068 errors 1000, some csum missing
root 5 inode 11825069 errors 1000, some csum missing
root 5 inode 11825070 errors 1000, some csum missing
root 5 inode 11825071 errors 1000, some csum missing
root 5 inode 11825072 errors 1000, some csum missing
root 5 inode 11825330 errors 1000, some csum missing
root 5 inode 11825359 errors 1000, some csum missing
found 900731358201 bytes used err is 1
total csum bytes: 1861763384
total tree bytes: 4889370624
total fs tree bytes: 1950982144
total extent tree bytes: 446197760
btree space waste bytes: 1016611646
file data blocks allocated: 1906566291456
 referenced 1920609484800
Btrfs v3.12

Although I used --repair, I have no idea if this repaired anything.

btrfs scrub ∞

Mounting worked like before. I tried to btrfs scrub, but it claims there's already one in progress. Checking its status, it's complete. I can't cancel it. What the fuck.

Thanks to the FAQ I got an easy answer:

perl -pi -e 's/finished:0/finished:1/' /var/lib/btrfs/*

unmounting and dmesg ∞

unmounting took fucking forever like before.

In the middle of the unmount attempt, I can see some interesting stuff in dmesg.

dmesg

[ 6591.629888] INFO: task btrfs-transacti:4266 blocked for more than 120 seconds.
[ 6591.629900]       Tainted: PF          O 3.13.0-32-generic #57-Ubuntu
[ 6591.629904] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 6591.629910] btrfs-transacti D ffff88013fc54440     0  4266      2 0x00000000
[ 6591.629920]  ffff8800641abdc0 0000000000000002 ffff8800a61c17f0 ffff8800641abfd8
[ 6591.629929]  0000000000014440 0000000000014440 ffff8800a61c17f0 ffff8801393c10a0
[ 6591.629936]  ffff8800ae0fc9e8 ffff8800ae0fc9e8 0000000000000000 ffff8800a904e780
[ 6591.629944] Call Trace:
[ 6591.629959]  [<ffffffff817200d9>] schedule+0x29/0x70
[ 6591.630017]  [<ffffffffa00e1def>] wait_current_trans.isra.21+0x9f/0x100 [btrfs]
[ 6591.630029]  [<ffffffff810aaf00>] ? prepare_to_wait_event+0x100/0x100
[ 6591.630067]  [<ffffffffa00e3348>] start_transaction+0x288/0x580 [btrfs]
[ 6591.630106]  [<ffffffffa00e36f7>] btrfs_attach_transaction+0x17/0x20 [btrfs]
[ 6591.630143]  [<ffffffffa00de810>] transaction_kthread+0x170/0x240 [btrfs]
[ 6591.630179]  [<ffffffffa00de6a0>] ? btrfs_cleanup_transaction+0x550/0x550 [btrfs]
[ 6591.630187]  [<ffffffff8108b3d2>] kthread+0xd2/0xf0
[ 6591.630195]  [<ffffffff8108b300>] ? kthread_create_on_node+0x1d0/0x1d0
[ 6591.630204]  [<ffffffff8172c5bc>] ret_from_fork+0x7c/0xb0
[ 6591.630211]  [<ffffffff8108b300>] ? kthread_create_on_node+0x1d0/0x1d0

re-mount and btrfs scrub ∞

After a couple of minutes, I decided to try to open another terminal and mount the faulty partition into another directory. It's taking forever.

After a long while the unmount worked. Then the remount worked.

Running another btrfs scrub gives:

scrub status for 84d22b45-b563-40ee-b7c7-be19887b74db
	scrub started at Sat Aug  9 00:03:44 2014 and finished after 28302 seconds
	total bytes scrubbed: 1.74TiB with 304 errors
	error details: read=304
	corrected errors: 111, uncorrectable errors: 193, unverified errors: 0

.. I see a lot of uncorrectable errors. But.. what does that even mean? What files are effected?

I couldn't boot into a system which attempted to mount that partition via fstab, even though I could manually mount it. Go figure.

I may have USB issues, as I had to move my keyboard and mouse to other motherboard ports to get them to work properly.

troubleshooting ideas, not pursued ∞

unmount, then maybe try..

\btrfs  check  --repair  /dev/sdb4

Further notes ∞

https://unix.stackexchange.com/questions/82996/understanding-btrfs-dmesg
check out btrfs-find-root <disk>
Then something like btrfs-restore -t 29360120 <disk> /mnt/

The hammer ∞

I ended up manually rsyncing bits and pieces from here and there, to rescue changes.

rsync picked up on i/o issues reading some files, and I copied that down into a list of files to re-download where possible.

I nuked the btrfs partition, formatted it ext4 and am restoring what I can. Fuck btrfs.

It turns out that the fuckup was originally using btrfs 0.20-rc1 and I was able to do some restoration with btrfs 3.12. Maybe I could continue to use the newer version, but fuck that. If I'm to migrate to BSD or another Linux distribution which doesn't support btrfs, then I don't want to go back to it anyways.

btrfs issues cause the hard drive to go insane ∞

Heat issues.

Issues booting on that drive.

During this time, one of my hard drives was predicting a 24-hour failure via smartctl -x <drive>. It turns out to not have been true, but because of heavy btrfs load the statistics were being skewed badly enough to trip alarms.

Still, now that I know how to read the extended smartctl information, I'm seeing pre-failure warnings.

High CPU usage and disk access with no tasks claiming responsibility is btrfs working behind the scenes. Naughty. This can be bad enough to trip smartctl warnings and cause significant system load.

Ancient links ∞

Defragment note ∞

Just dropping this note here:

\smartctl  -t long   /dev/sdx
\sudo  \nice -n 12  \btrfs  filesystem  defragment  -c -r -f /mnt/mnt/ -v

  • I didn't defragment a different partition because of some ioctl issue

Comment

Leave a Reply

Your email address will not be published. Required fields are marked *