AXN10 Woes

I bought a Cambridge Audio AX system for my office because it was cheap and it let me play records too. It’s really not too bad but I have my suspicions over the AXN10 Network Streamer. I’ve noticed it has a tendency to stop playing after a while. It plays fine for a good four to six months and then it’s like it runs out of buffer space or something and can only play music in five second chunks followed by a five second pauses. It’s peculiar. It’s also done it three times now so it’s now something that needs investigating.

My current setup uses my music stores on a WD MyCloud Mirror (that also needs fixing) that is streamed out via AssetUPnP running on an Ubuntu 22.04.5 LTS box to my CXNv2 and my AXN10. I originally used Twonky but Twonky is now, effectively, a dead platform. It doesn’t and never will support the Disk Number fields so multi-disk albums are a problem with Twonky.

I was hoping that I could use AssetUPnP to log the music and the duration played but I’m not convinced it’s logging is that capable. It seemed to log connections from the TV but not from the CA AXN which is a bit of a problem when I’m interested in what the AXN is doing.

The AXN itself doesn’t seem to do any logging either but, obviously, the Stream Magic app connects over APIs to the device so I was wondering whether there was a hidden interface there that I could poke. I’m really not the best when it comes to reverse engineering interfaces so I did something really dumb and just put in the IP address of the AXN into a browser followed by /debug. Surprisingly, this returned a “Remote debug not enabled” message which immediately made me think that there must be a way to enable remote debug. Time to go digging…

Ecosia wasn’t really of much help here but Google, especially Googles AI bot was most helpful. Initially it pointed me to a very nice project on GitHub (https://github.com/noahhusby/aiostreammagic)

Now, I’m no python developer (I’m no developer at all, really) but by explaining what I was trying to achieve Google came up with the following program:

import asyncio
import csv
import os
import sys
from datetime import datetime
from aiostreammagic import StreamMagicClient

# CONFIGURATION
AXN10_IP = "IP GOES HERE"  # Replace with your exact AXN10 IP
LOG_FILE = "/usr/local/bin/axn10-tracker/streamer_playback_log.csv"

class PlaybackTracker:
    def __init__(self):
        self.current_track = None
        self.current_artist = None
        self.play_start_time = None
        self.last_state = None

    def log_to_csv(self, start_time, stop_time, artist, track, duration_seconds):
        """Appends a completed playback record to the CSV file."""
        file_exists = os.path.isfile(LOG_FILE)
        
        try:
            with open(LOG_FILE, mode='a', newline='', encoding='utf-8') as f:
                writer = csv.writer(f)
                if not file_exists:
                    writer.writerow(["Start Time", "Stop Time", "Artist", "Track", "Played Duration (Sec)", "Played Duration (Min)"])
                
                duration_minutes = round(duration_seconds / 60, 2)
                writer.writerow([
                    start_time.strftime("%Y-%m-%d %H:%M:%S"),
                    stop_time.strftime("%Y-%m-%d %H:%M:%S"),
                    artist or "Unknown Artist",
                    track or "Unknown Track",
                    round(duration_seconds, 1),
                    duration_minutes
                ])
            print(f"Logged to CSV: '{track}' by {artist} - {duration_minutes} mins.", flush=True)
        except Exception as e:
            print(f"Failed writing to CSV: {e}", file=sys.stderr, flush=True)

    async def on_state_change(self, client: StreamMagicClient, *args):
        """Official callback engine triggered immediately on AXN10 updates."""
        # Grab the core play state object safely
        ps_obj = client.play_state
        if ps_obj:
            # Extract simple state string (e.g., 'play', 'pause', 'stop', 'idle')
            play_state = getattr(ps_obj, 'state', 'stopped').lower()
            
            # Extract nested metadata objects
            metadata = getattr(ps_obj, 'metadata', None)
            track_title = getattr(metadata, 'title', None) if metadata else None
            artist_name = getattr(metadata, 'artist', None) if metadata else None
        else:
            play_state = "stopped"
            track_title = None
            artist_name = None

        # Clean diagnostic line showing exactly what song has been processed
        print(f"[TRACKER STATUS] State: {play_state.upper()} | Title: '{track_title}' by {artist_name}", flush=True)

        now = datetime.now()

        # CASE 1: Streamer just entered a play cycle
        if play_state == "play" and self.last_state != "play":
            self.play_start_time = now
            self.current_track = track_title
            self.current_artist = artist_name
            self.last_state = "play"
            print(f"Playback started: '{track_title}' by {artist_name}", flush=True)

        # CASE 2: Tracks shifted during active streaming
        elif play_state == "play" and self.current_track != track_title:
            if self.play_start_time:
                duration = (now - self.play_start_time).total_seconds()
                if duration > 2:
                    self.log_to_csv(self.play_start_time, now, self.current_artist, self.current_track, duration)
            
            self.play_start_time = now
            self.current_track = track_title
            self.current_artist = artist_name
            print(f"Track changed: '{track_title}' by {artist_name}", flush=True)

        # CASE 3: Playback stopped, paused, or returned to idle
        elif play_state in ["pause", "stop", "idle"] and self.last_state == "play":
            if self.play_start_time:
                duration = (now - self.play_start_time).total_seconds()
                if duration > 2:
                    self.log_to_csv(self.play_start_time, now, self.current_artist, self.current_track, duration)
            self.play_start_time = None
            self.current_track = None
            self.last_state = play_state
            print(f"Playback stopped ({play_state}).", flush=True)

async def main():
    tracker = PlaybackTracker()
    print(f"Initializing event pipeline for AXN10 at {AXN10_IP}...", flush=True)
    
    while True:
        try:
            client = StreamMagicClient(AXN10_IP)
            
            # 1. Bind our handler logic block
            await client.register_state_update_callbacks(tracker.on_state_change)
            
            # 2. ACTUALLY EXECUTE THE NETWORK HANDSHAKE CONNECTION
            await client.connect()
            
            print("Connected! Active socket listening for streaming metrics...", flush=True)
            
            # Keep the context block alive indefinitely
            while True:
                await asyncio.sleep(3600)
                
        except Exception as e:
            print(f"Network drop encountered: {e}. Reconnecting in 15 seconds...", file=sys.stderr, flush=True)
            await asyncio.sleep(15)

if __name__ == "__main__":
    try:
        asyncio.run(main())
    except KeyboardInterrupt:
        print("\nShutdown signal captured.")

This logs every event to a CSV and lets me keep track of everything that has been played and for how long. It also came up with a companion program to summarise the data for me. I probably could have done that bit myself but I was more interested in having something work that hacking around trying to make things work.

import csv
from datetime import datetime
from collections import defaultdict

LOG_FILE = "/usr/local/bin/axn10-tracker/streamer_playback_log.csv"

def analyze_logs():
    total_seconds = 0
    track_counts = defaultdict(int)
    track_durations = defaultdict(float)
    
    try:
        with open(LOG_FILE, mode='r', encoding='utf-8') as f:
            reader = csv.DictReader(f)
            
            for row in reader:
                # Calculate overall cumulative time
                seconds = float(row["Played Duration (Sec)"])
                total_seconds += seconds
                
                # Group data by track identifier
                track_identifier = f"'{row['Track']}' by {row['Artist']}"
                track_counts[track_identifier] += 1
                track_durations[track_identifier] += seconds
                
    except FileNotFoundError:
        print(f"Error: Could not find '{LOG_FILE}'. Make sure the logger script has run and recorded at least one song.")
        return

    # Convert total seconds into hours and minutes
    total_minutes = total_seconds / 60
    total_hours = total_minutes / 60
    
    print("=" * 50)
    print("      CAMBRIDGE AUDIO AXN10 PLAYBACK REPORT      ")
    print("=" * 50)
    print(f"Total Listening Time: {total_hours:.2f} Hours ({int(total_minutes)} Minutes)")
    print("-" * 50)
    
    # Sort and find the tracks listened to for the longest total duration
    print("Top 5 Most Listened-To Tracks (By Actual Duration):")
    sorted_by_duration = sorted(track_durations.items(), key=lambda x: x[1], reverse=True)
    
    for i, (track, seconds) in enumerate(sorted_by_duration[:5], 1):
        mins = seconds / 60
        print(f" {i}. {track} — Listened for {mins:.1f} mins total")

if __name__ == "__main__":
    analyze_logs()

The next thing we worked on was making it into a service so that it would always run. The last thing I needed was to forget to log a week or month’s worth of data and complete stuff my numbers up. Adding it as a service was pretty trivial.

First create the logger service:

sudo vi /etc/systemd/system/axn10-logger.service

   [Unit]
   Description=Cambridge Audio AXN10 Playback Logger
   After=network-online.target
   Wants=network-online.target

   [Service]
   Type=simple
   User=YOUR_UBUNTU_USERNAME
   WorkingDirectory=/usr/local/bin/axn10-tracker
   ExecStart=/usr/local/bin/axn10-tracker/venv/bin/python3 /usr/local/bin/axn10-tracker/axn10_logger.py
   Restart=always
   RestartSec=10
   StandardOutput=journal
   StandardError=journal

   [Install]
   WantedBy=multi-user.target

then enable and run the service:

sudo systemctl daemon-reload
sudo systemctl enable axn10-logger.service
sudo systemctl start axn10-logger.service

It actually took a few iterations to get the logging code to properly run as a service and to log the data correctly but Google helped me achieve something I could not have done on my own.

So now, every time I play something on my AXN it gets logged to a file and over time it will show me, not only the total play time between failures, it will also show me what artists / albums / songs I’m listening to the most. That’s pretty cool.

MakeMKV

Under the Advanced settings it is possible to tweak what MakeMKV is looking for when it rips a disc.

The original string is:

-sel:all,+sel:(favlang|nolang),-sel:mvcvideo,=100:all,-10:favlang

Which basically selects favourite language files and no language files, and deselects mvcvideo (3D) files. No idea what the bit on the end does but it does explain why my 3D films never worked properly.

From looking on the MakeMKV forum I’m now using:

-sel:all,+sel:(favlang|nolang|mvcvideo),=100:all,-10:favlang

which should let me rip 3D films and, now that I’ve sent falling to English, only selects the English tracks.

Plex and Subtitles – 2

Subtitle Editor seems to be the go-to program for editing subtitles. However, being on a Mac, that’s no good for me as there are only Windows and Linux versions. The thing with having a Linux version of a tool though is that there’s a very good chance that it exists in Docker and that is, indeed, the case here.

https://hub.docker.com/r/fresta/subtitleedit

docker run --name=subtitleedit -e WIDTH=1280 -e HEIGHT=720 -p 3389:3389 -p 8080:8080 -v /media:/media fresta/subtitleedit

Just set /media to be something useful.

Plex and Subtitles

I’ve spent far too much time ripping all my Blu-Rays to make sure they’ve got the full compliment of English audio and subtitle tracks. However, that might have been the wrong way of doing things. I mean, I don’t really care all that much about commentary tracks at all, I just wanted them there for the sake of completeness.

Now, the other day we were watching The Mummy, the rather excellent film with Brendan Fraser and Rachel Weisz and I was getting rather irritated because the subtitles I was expecting to see simply weren’t there. I’m pretty sure there were there before I started my “big rip” programme but they’re not there now. Well, that’s just annoying.

I started to cycle through the various subtitle tracks and not one of them was just the forced subtitles. It was everything or nothing. That was rather irritating too.

I re-ripped the disc looking for a subtitle track I might have missed but, alas, no. I’d got them all. I tried ripping just the tracks listed as “forced” but, if I took just those, then no subtitles were present in the final mkv. Selecting the subtitle track before the forced track gave me everything.

Also, The Mummy contains TextST tracks which, when ripped, completely break IINA for some reason. It turns out that The Mummy simply doesn’t have any forced subtitles at all. The Mummy Returns does, they’re easy to find, but not The Mummy.

On the MakeMKV forums I found a nice little snippet of helpful code:

mkvmerge -i <MKV File>
mkvextract tracks <MKV File> <track>:<export subtitle>

So, for The Mummy the first part looks like this:

mkvmerge -i The\ Mummy\ \(1999\).mkv 
File 'The Mummy (1999).mkv': container: Matroska
Track ID 0: video (VC-1)
Track ID 1: audio (DTS-HD Master Audio)
Track ID 2: audio (DTS)
Track ID 3: audio (AC-3)
Track ID 4: audio (AC-3)
Track ID 5: audio (AC-3)
Track ID 6: subtitles (HDMV PGS)
Track ID 7: subtitles (HDMV PGS)
Chapters: 18 entries

This shows that tracks 6 and 7 are the subtitle tracks. Now, at this point we could play the video in IINA and see which one of the two is the better extract possibility. Doing that we can see that track 6 aligns with the speech and track 7 appears to be the commentary. So, that means, we want track 6.

If we’re going to create an external subtitle file we really want our subtitles to be in an SRT format so we’ll name the file accordingly. The extract might take a while, especially if the extract is taking place over the network.

 mkvextract tracks The\ Mummy\ \(1999\).mkv 6:English_1.srt
Extracting track 6 with the CodecID 'S_HDMV/PGS' to the file 'English_1.sup'. Container format: SUP
Progress: 100%

We can see that the container format is actually SUP and not SRT so the file should be renamed to reflect that.

mv English_1.srt English_1.sup

Now, somehow, we just need to convert the SUP to SRT. Fortunately, there’s a website that can help with that:

https://subtitletools.com

The problem now is that the file just downloaded is the whole of the subtitle track which means that it is now necessary to go through the entire file and rip out all the entries that are not needed. It’s a bit of a painful process but, fortunately, in the case of The Mummy, it is possible to use “Egyptian” as a marker for where to look for the forced subtitles.

Once we have a file that looks reasonable, we just have to give it a sensible name, for Plex this looks like:

The Mummy (1999).eng.forced.srt

And, once that is done, The Mummy now has forced subtitles in all the right places. Well, not “all” the right places because the source file still appears to be missing the occasional entry but it’s a good as it’s going to get.

Hello Twonky! This is Europe calling.

I’m still looking at getting the album order correct on multi-disc albums. Only now, I’m running Twonky on my T12-423 rather than the WD NAS so I can pretty much do what I want to now.

So, first things first. The Twonky calls.

I’m using HTTPie but Postman would be equally good. First, we need to construct a post to read the available albums:

POST http://192.168.3.16:9000/dev0/srv1/control

<s:Envelope s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Body>
<u:Browse xmlns:u="urn:schemas-upnp-org:service:ContentDirectory:1">
<ObjectID>0$1$12</ObjectID>
<BrowseFlag>BrowseDirectChildren</BrowseFlag>
<Filter>*</Filter>
<StartingIndex>233</StartingIndex>
<RequestedCount>300</RequestedCount>
<SortCriteria/>
</u:Browse>
</s:Body>
</s:Envelope>

I can’t remember where the original object ID comes from (a thing for another day) but the StartingIndex and the RequestedCount are where you want to start in your list of albums and how many albums you’d like returned. Twonky only seemed to return 232 which is why I started at 233. I deliberately kept the RequestCount High because Twonky will tell me how many entries its retuned so I’m not too fussed about being accurate.

Twonky returns coded XML which looks like this:

<DIDL-Lite xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:upnp="urn:schemas-upnp-org:metadata-1-0/upnp/" xmlns:dlna="urn:schemas-dlna-org:metadata-1-0/" xmlns:arib="urn:schemas-arib-or-jp:elements-1-0/" xmlns:dtcp="urn:schemas-dtcp-com:metadata-1-0/" xmlns:pv="http://www.pv.com/pvns/" xmlns="urn:schemas-upnp-org:metadata-1-0/DIDL-Lite/"><container id="0$1$12$4282" parentID="0$1$12" restricted="1" childCount="38" searchable="1"><dc:title>Eurovision Song Contest 2014: Copenhagen - Join Us</dc:title><upnp:genre>Pop</upnp:genre><upnp:album>Eurovision Song Contest 2014: Copenhagen - Join Us</upnp:album><dc:creator>Various Artists</dc:creator><upnp:albumArtURI dlna:profileID="JPEG_TN" >http://192.168.3.16:9000/disk/DLNA-PNJPEG_TN-OP01-CI1-FLAGS00d00000/defaa/C/O0$1$12$4282.jpg?scale=org</upnp:albumArtURI><pv:childCountContaine

Copy the body into SublimeText and find / replace the &lt; and &gt; with < and > respectively, then indent the XML to get a nicely formatted page. From this I can easily see that the object for Eurovision 2024 is:

<container childCount="37" dc:date="2024-01-01T12:00:00" id="0$1$12$5307" parentID="0$1$12" restricted="1" searchable="1">
<dc:title>Eurovision Song Contest 2024: Malmö - United By Music</dc:title>
<dc:date>2024-01-01</dc:date>
<upnp:genre>Pop</upnp:genre>
<upnp:album>Eurovision Song Contest 2024: Malmö - United By Music</upnp:album>
<dc:creator>Various Artists</dc:creator>
<upnp:albumArtURI dlna:profileID="JPEG_TN">http://192.168.3.16:9000/disk/DLNA-PNJPEG_TN-OP01-CI1-FLAGS00d00000/defaa/C/O0$1$12$5307.jpg?scale=org</upnp:albumArtURI
<pv:childCountContainer>0</pv:childCountContainer>
<upnp:artist>Various Artists</upnp:artis>
<pv:modificationTime>421989616</pv:modificationTime><pv:containerContent>object.item.audioItem.musicTrack</pv:containerContent>
<upnp:class>object.container.album.musicAlbum</upnp:class>
</container>

Now, make a call to Twonky using the same parameters as the CXN makes:

POST http://192.168.3.16:9000/dev0/srv1/control
<s:Envelope s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Body>
<u:Browse xmlns:u="urn:schemas-upnp-org:service:ContentDirectory:1">
<ObjectID>0$1$12$5307</ObjectID>
<BrowseFlag>BrowseDirectChildren</BrowseFlag>
<Filter>dc:title,upnp:class,upnp:album,upnp:originalTrackNumber,id,res,res@protection,res@duration,upnp:searchClass,upnp:artist,upnp:genre,upnp:albumArtURI</Filter>
<StartingIndex>0</StartingIndex>
<RequestedCount>100</RequestedCount>
<SortCriteria>+upnp:originalTrackNumber,+dc:title</SortCriteria>
</u:Browse>
</s:Body>
</s:Envelope>

This returns:

<?xml version="1.0" encoding="utf-8"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<s:Body>
<u:BrowseResponse xmlns:u="urn:schemas-upnp-org:service:ContentDirectory:1">
<Result>
<DIDL-Lite xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:upnp="urn:schemas-upnp-org:metadata-1-0/upnp/" xmlns:dlna="urn:schemas-dlna-org:metadata-1-0/" xmlns:arib="urn:schemas-arib-or-jp:elements-1-0/" xmlns:dtcp="urn:schemas-dtcp-com:metadata-1-0/" xmlns:pv="http://www.pv.com/pvns/" xmlns="urn:schemas-upnp-org:metadata-1-0/DIDL-Lite/">
<item id="0$1$12$5307R891915" parentID="0$1$12$5307" restricted="1">
<dc:title>Hurricane [Israel]</dc:title>
<upnp:genre>Pop</upnp:genre>
<upnp:album>Eurovision Song Contest 2024: Malmö - United By Music</upnp:album>
<upnp:originalTrackNumber>1</upnp:originalTrackNumber>
<upnp:albumArtURI dlna:profileID="JPEG_TN" >http://192.168.3.16:9000/disk/DLNA-PNJPEG_TN-OP01-CI1-FLAGS00d00000/defaa/A/O0$1$8I891915.jpg?scale=org</upnp:albumArtURI>
<upnp:artist>Eden Golan</upnp:artist>
<res duration="0:03:01.440" protocolInfo="http-get:*:audio/x-aiff:*" >http://192.168.3.16:9000/disk/NON-DLNA-OP01-FLAGS01700000/O0$1$8I891915.aif</res>
<upnp:class>object.item.audioItem.musicTrack</upnp:class>
</item>
<item id="0$1$12$5307R887307" parentID="0$1$12$5307" restricted="1">
<dc:title>Titan [Albania]</dc:title>
<upnp:genre>Pop</upnp:genre>
<upnp:album>Eurovision Song Contest 2024: Malmö - United By Music</upnp:album>
<upnp:originalTrackNumber>1</upnp:originalTrackNumber>
<upnp:albumArtURI dlna:profileID="JPEG_TN" >http://192.168.3.16:9000/disk/DLNA-PNJPEG_TN-OP01-CI1-FLAGS00d00000/defaa/A/O0$1$8I887307.jpg?scale=org</upnp:albumArtURI>
<upnp:artist>BESA</upnp:artist>
<res duration="0:02:58.226" protocolInfo="http-get:*:audio/x-aiff:*" >http://192.168.3.16:9000/disk/NON-DLNA-OP01-FLAGS01700000/O0$1$8I887307.aif</res>
<upnp:class>object.item.audioItem.musicTrack</upnp:class>
</item>

Hurricane is not the first track for the album so this is obviously wrong.

As I can’t change what the CXN is asking for, I need to change the response the CXN is receiving (if that is even possible?)

What I think I need to do is to have the “upnp:originalTrackNumber” be a combination of the disc number and the actual track number so it becomes something like 101 and 201 for the first track on the first and second disks… I think I’m out of luck there.

Expanding a RAID – Part Deux

Make sure the filesystem isn’t being used:

sudo systemctl stop plexmediaserver
sudo systemctl stop smbd

unmount the filesystem

sudo umount /storage

use parted to resize the filesystem

sudo parted
resize part 1 100%

remount

sudo mount /storage

resize the Btrfs filesystem

sudo btrfs filesystem resize max /storage

and, all being well, the system isn’t hosed.

Expanding a RAID

Quick overview more for me than anything else:

lsblk to see disks in system:

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 9.1T 0 disk
└─md3 9:3 0 27.3T 0 raid5
└─md3p1 259:0 0 27.3T 0 part /storage
sdb 8:16 0 9.1T 0 disk
└─md3 9:3 0 27.3T 0 raid5
└─md3p1 259:0 0 27.3T 0 part /storage
sdc 8:32 0 3.6T 0 disk
├─sdc1 8:33 0 1G 0 part
├─sdc2 8:34 0 1G 0 part
│ └─md0 9:0 0 1022M 0 raid1
│ └─md0p1 259:2 0 1020M 0 part /boot
├─sdc3 8:35 0 128G 0 part
│ └─md1 9:1 0 127.9G 0 raid1
│ └─md1p1 259:1 0 127.9G 0 part [SWAP]
└─sdc4 8:36 0 3.5T 0 part
└─md2 9:2 0 3.5T 0 raid1
└─md2p1 259:3 0 3.5T 0 part /
sdd 8:48 0 3.6T 0 disk
├─sdd1 8:49 0 1G 0 part /boot/efi
├─sdd2 8:50 0 1G 0 part
│ └─md0 9:0 0 1022M 0 raid1
│ └─md0p1 259:2 0 1020M 0 part /boot
├─sdd3 8:51 0 128G 0 part
│ └─md1 9:1 0 127.9G 0 raid1
│ └─md1p1 259:1 0 127.9G 0 part [SWAP]
└─sdd4 8:52 0 3.5T 0 part
└─md2 9:2 0 3.5T 0 raid1
└─md2p1 259:3 0 3.5T 0 part /
sde 8:64 0 9.1T 0 disk
└─md3 9:3 0 27.3T 0 raid5
└─md3p1 259:0 0 27.3T 0 part /storage
sdf 8:80 0 9.1T 0 disk
└─md3 9:3 0 27.3T 0 raid5
└─md3p1 259:0 0 27.3T 0 part /storage

add in new disk and lsblk again:

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 9.1T 0 disk
└─md3 9:3 0 27.3T 0 raid5
└─md3p1 259:0 0 27.3T 0 part /storage
sdb 8:16 0 9.1T 0 disk
└─md3 9:3 0 27.3T 0 raid5
└─md3p1 259:0 0 27.3T 0 part /storage
sdc 8:32 0 3.6T 0 disk
├─sdc1 8:33 0 1G 0 part
├─sdc2 8:34 0 1G 0 part
│ └─md0 9:0 0 1022M 0 raid1
│ └─md0p1 259:2 0 1020M 0 part /boot
├─sdc3 8:35 0 128G 0 part
│ └─md1 9:1 0 127.9G 0 raid1
│ └─md1p1 259:1 0 127.9G 0 part [SWAP]
└─sdc4 8:36 0 3.5T 0 part
└─md2 9:2 0 3.5T 0 raid1
└─md2p1 259:3 0 3.5T 0 part /
sdd 8:48 0 3.6T 0 disk
├─sdd1 8:49 0 1G 0 part /boot/efi
├─sdd2 8:50 0 1G 0 part
│ └─md0 9:0 0 1022M 0 raid1
│ └─md0p1 259:2 0 1020M 0 part /boot
├─sdd3 8:51 0 128G 0 part
│ └─md1 9:1 0 127.9G 0 raid1
│ └─md1p1 259:1 0 127.9G 0 part [SWAP]
└─sdd4 8:52 0 3.5T 0 part
└─md2 9:2 0 3.5T 0 raid1
└─md2p1 259:3 0 3.5T 0 part /
sde 8:64 0 9.1T 0 disk
└─md3 9:3 0 27.3T 0 raid5
└─md3p1 259:0 0 27.3T 0 part /storage
sdf 8:80 0 9.1T 0 disk
└─md3 9:3 0 27.3T 0 raid5
└─md3p1 259:0 0 27.3T 0 part /storage
sdg 8:96 0 9.1T 0 disk
├─sdg1 8:97 0 2G 0 part
│ └─md127 9:127 0 0B 0 md
├─sdg2 8:98 0 9.1T 0 part
│ └─md126 9:126 0 0B 0 md
├─sdg3 8:99 0 1G 0 part
└─sdg4 8:100 0 1G 0 part

/dev/sdg has been added

However, because sdg is showing as md127 we need to stop and remove the raid:

mdadm --stop /dev/md127
mdadm --remove /dev/md127

now we can partition the disk

use parted to wipe filesystem and add a new label to the disk

parted -s -a optimal /dev/sdg mklabel gpt

Use fdisk to check the new GTP Partition tables. Don’t bother to create any partitions.

fdisk -p to check out the unpartitioned disk:

$ sudo fdisk /dev/sdg
Welcome to fdisk (util-linux 2.39.3).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): p
Disk /dev/sdg: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Disk model: WDC WD100EFAX-68
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 56FF5A1F-FE90-4AB1-ABD6-DBB1E1D22042

Command (m for help): q

Just check to see which array you’re adding the disk to:

$ cat /prod/mdstat

Personalities : [raid6] [raid5] [raid4] [raid1] [raid0] [raid10]
md1 : active raid1 sdd3[1] sde3[0]
134085632 blocks super 1.2 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk

md2 : active raid1 sdd4[1] sde4[0]
3770516480 blocks super 1.2 [2/2] [UU]
bitmap: 1/29 pages [4KB], 65536KB chunk

md0 : active raid1 sdd2[1] sde2[0]
1046528 blocks super 1.2 [2/2] [UU]

md3 : active raid5 sdf[2] sdb[1] sdg[4] sda[0]
29298914304 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 0/73 pages [0KB], 65536KB chunk

unused devices: <none>

md3 in this case. Add the disk into the array:

mdadm --add /dev/md3 /dev/sdg

Check the array:

$ sudo mdadm --add /dev/md3 /dev/sdg
mdadm: added /dev/sdg

sysadmin@tnas:~$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1] [raid0] [raid10]
md1 : active raid1 sdd3[1] sde3[0]
134085632 blocks super 1.2 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk

md2 : active raid1 sdd4[1] sde4[0]
3770516480 blocks super 1.2 [2/2] [UU]
bitmap: 2/29 pages [8KB], 65536KB chunk

md0 : active raid1 sdd2[1] sde2[0]
1046528 blocks super 1.2 [2/2] [UU]

md3 : active raid5 sdc5 sdf[2] sdb[1] sdg[4] sda[0]
29298914304 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
bitmap: 0/73 pages [0KB], 65536KB chunk

The disk is now visible in the array: sdc5 but it’s marked as a spare disk (S).

Now we need to grow the array:

mdadm --grow /dev/md3 --raid-devices=5

where the number of raid devices is the number of disks we want in the raid.

Watch /proc/mdstat to see the progress:

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1] [raid0] [raid10]
md1 : active raid1 sdd3[1] sde3[0]
134085632 blocks super 1.2 [2/2] [UU]
bitmap: 0/1 pages [0KB], 65536KB chunk

md2 : active raid1 sdd4[1] sde4[0]
3770516480 blocks super 1.2 [2/2] [UU]
bitmap: 1/29 pages [4KB], 65536KB chunk

md0 : active raid1 sdd2[1] sde2[0]
1046528 blocks super 1.2 [2/2] [UU]

md3 : active raid5 sdc[5] sdf[2] sdb[1] sdg[4] sda[0]
29298914304 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]
[>………………..] reshape = 0.0% (337408/9766304768) finish=3376.6min speed=48201K/sec
bitmap: 0/73 pages [0KB], 65536KB chunk

unused devices: <none>

Yep, this is going to take a while. Time to put the kettle on.

Helpful guides:

https://wpguru.co.uk/2021/01/expand-software-raid-mdadm/
https://www.itsfullofstars.de/2019/03/how-to-add-a-new-disk-to-raid5/
https://tomlankhorst.nl/setup-lvm-raid-array-mdadm-linux

MTU Sizing

Edit /etc/netplan/50-cloud-init.yaml

network:
ethernets:
enp3s0:
dhcp4: true
dhcp6: true
mtu: 1472
version: 2

Then do:

sudo netplan try

and accept the change if it seems to work.

  • the default frame size is 1500.

Ubuntu 24.04 on a Terra-Master T12-423

My reasons for putting Ubuntu are explained here: Confession Time and I’m using Ubuntu because that’s what Terra-Master use themselves so it should work just fine and it’s the distribution I’m slightly more familiar with at the moment. Debian would be my alternative choice.

First things first. Open up the case and remove the small SanDisk USB stick. This contains the TOS boot loader and we don’t want that anymore. Go and put it in a marked envelope in a drawer somewhere just in case you ever want to put it back.

Step 2 – not for the faint of heart. Update the BIOS. The full thread is here: https://forum.terra-master.com/en/viewtopic.php?t=4238 and with that done the rest is pretty straightforward.

Next we need to change the boot order. With the TOS USB stick removed that’s an awful lot easier. For some reason, for me at least, the TOS USB stick always took priority no matter what I did so with that out of the way it’s a lot easier to do things now.

I actually burned Ubuntu to DVD rather than a flash drive simply because I don’t think the flash drives I have are very reliable. They are incredibly slow for some reason too. So, I opted for DVD.

Slap the DVD in and run through the basic setup until the drive configuration page presents itself. Here choose the custom layout option. I have two Seagate IronWolf disks as my boot disks so I wanted a mirrored raid boot setup. Not having done that before I found a very easy to follow video (and I really hate having to follow videos) here:

Installing Ubuntu 24.04 With RAID

and a really useful page here:

Ubuntu 24.04 with software RAID1 and UEFI

The only changes to the process I made was to include in my four 10TB raid disks into the process. That means that at the end of the video I have a filesystem that looks something like:

md0 RAID1 /boot
md1 RAID1 SWAP
md2 RAID1 /root
md3 RAID5 /storage

It also means that as the storage is built at installation time I can just sit back and let the RAID configure itself for a few hours.

Because I always choose a minimised installation I need to add a few things in to make the environment usable so this is what I do:

sudo apt update
sudo apt install htop
sudo apt install avahi-daemon
sudo apt install vim
sudo apt install cron
sudo apt upgrade

Once cron and vim are installed I set up my lazy mdstat rebuild page.

I have a little web server with a basic html page of:

<!DOCTYPE html>
<html>
<head>
<title>
RAID Rebuild Status
</title>
<meta http-equiv="refresh" content="60" >
</head>
<body>
<embed src="raid.txt" width="100%" height="1000">
</body>
</html>

It’s really not much of a page. Then on the T12 I add in a cron job:

* * * * * cat /proc/mdstat | ssh ricardo@server.local 'cat - > /home/ricardo/docker/dumkid/html/raid.txt'

This means that I can point my phone / laptop / work computer to my little webpage and see how the rebuild is going without having to log in to the T12 itself.