At Hurricane Labs, we love logs and we love video games. Any chance to combine the two is exciting, and recently I stumbled on just such an opportunity.
My friends and I have been hooked on 7 Days to Die, a survival horror game in which players scavenge an apocalypse-ridden world and build a defense to survive a massive zombie horde that attacks every seven days. One of my friends had found an itema book, specificallythat wouldve been a significant upgrade for another player, but when they met up to exchange the item it had disappeared. We dismissed it as a glitch, but I began to wonderdid the game have logs, and could they help us discover what had happened? Naturally, my weapon of choice was Splunk.
Server-side logs are actually pretty rare in the gaming world these days. The concept of hosting a local server is antiquated, replaced by online servers hosted by game companies themselves. 7 Days to Die, however, allows players to operate their own server and configure it however theyd like. We didnt realize this at firstmy brother was the first one to start the game and we all joined him, unaware that we were actually joining a server running locally on his computer, which also happened to be the one with the least amount of resources. Eventually he sent me the server files and I started hosting from mine.
I was pleasantly surprised by the logs that had been generated. Every log starts with a consistent and readable timestamp and a generally uniform message format. For example:
Copy to Clipboard
2021-01-20T23:31:49 14350.539 INF GMSG: Player xAweSamx left the game
Scrolling through the unexpectedly large file, I noticed that in addition to operational logs, there were logs about player behavior and AI behavior: kills, deaths and even how many zombies were spawned! It was time to get these log files into Splunk and really start digging in.
I chose to use 7daystodie:output as the custom sourcetype name to reflect both the software and the log file name (output_log) in case there is ever another log filetype we want to ingest.
The first thing I noticed is that every log file begins with a header that has interesting information but isnt something we want to index every time:
Copy to Clipboard
Mono path[0] = D:/SteamLibrary/steamapps/common/7 Days To Die/7daystodie_Data/ManagedMono config path = D:/SteamLibrary/steamapps/common/7 Days To Die/MonoBleedingEdge/etcInitialize engine version: 2019.2.17f1 (8e603399ca02)[XR] Discovering subsystems at path D:/SteamLibrary/steamapps/common/7 Days To Die/7daystodie_Data/UnitySubsystemsForcing GfxDevice: Direct3D 11GfxDevice: creating device client; threaded=1Direct3D:Version: Direct3D 11.0 [level 11.1] Renderer: NVIDIA GeForce RTX 2070 SUPER (ID=0x1e84)Vendor:VRAM: 8011 MBDriver: 27.21.14.6079Begin MonoManager ReloadAssembly Completed reload, in 2.962 seconds Initializing input.
Input initialized.
Initialized touch support.
UnloadTime: 6.034400 ms
To omit this, I took advantage of the FIELD_HEADER_REGEX props setting. This uses Regular Expression to identify the last line of the header and in this case tells Splunk not to index anything above it:
Copy to Clipboard
[7daystodie:output]FIELD_HEADER_REGEX = UnloadTime
Line-breaking and timestamp parsing was the next objective. Fortunately, it uses a pretty standard and consistent timestamp, which is already more than I can say for some other more common data sources. When I first loaded this into the data preview through the web interface, Splunk actually did a good job identifying and separating events without any further configuration. The best practice though is to be as specific as possible here to avoid any potential issues now or in the future.
At Hurricane Labs we like to cover at least TIME_PREFIX, TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD, and TIMEZONE. Between these settings, weve specified where in the log the timestamp occurs, how far from that point to consider part of the timestamp, what the timestamp looks like, and what timezone the log was written in. We also prefer to have control over the event line-breaking, in other words specifying what constitutes a single event. In most cases, such as ours, the event in the log begins with a timestamp so weve told Splunk not to line merge automatically and we specify our own condition in the LINE_BREAKER setting.
As youll recall from the brief example above, our timestamp looks like this:
Copy to Clipboard
2021-01-20T23:31:49
Our props configuration so far is:
Copy to Clipboard
[7daystodie:output]FIELD_HEADER_REGEX = UnloadTimeLINE_BREAKER = ([rn]+)d{4}-d{2}-d{2}Td{2}:d{2}:d{2}MAX_TIMESTAMP_LOOKAHEAD = 19SHOULD_LINEMERGE = falseTIMEZONE = US/EasternTIME_FORMAT = %Y-%m-%dT%H:%M:%STIME_PREFIX = ^
I ingested a single log file to a test index to make sure everything behaved as expected and started playing with the data to see what was available. Splunk automatically extracted some fields that were already key=value pairs, but some more specific field extractions were necessary. I had data about when players connected and disconnected, how long their session was, how many zombies (and what types) they killed, and if they died. I also had some internal system messages such as audio files failing to load, no space to load in a resource, andmost relevant to our original goal of discovering the fate of our missing itemwhen a resource fell off the world.
Some examples of the field extractions:
Copy to Clipboard
EXTRACT-playerinfo = INFs+Players+(?.+?)s+disconnecteds+afters+(?d+.d+)s+minutesEXTRACT-spawnerinfo = INFs+(?S+)s+spawneds+(?S+)EXTRACT-vendorprio = d{4}-d{2}-d{2}Td{2}:d{2}:d{2}s+d+.d+s+(?S+)s+EXTRACT-killerinfo = Entitys+(?S+)s+d+s+killeds+bys+(?S+)s+EXTRACT-component = d{4}-d{2}-d{2}Td{2}:d{2}:d{2}s+d+.d+s+WRNs+(?NosS+|D+?)s+
If youre not familiar with reading Regular Expression one of the above extractions pulls out the priority: INF (informational) or WRN (warning). Narrowing down on the warning messages I began to see events such as:
Copy to Clipboard
2021-01-24T22:36:46 8357.174 WRN Entity [type=EntityItem, name=resourceScrapIron, cnt=150] fell off the world, id=51777 pos=(1327.2, -0.5, 108.7)
Fell off the world is an interesting fate for an item, but the game is technically in early access (meaning it is still being worked on). Honestly, open world survival games usually have bugs similar to these simply due to the scope of the environment and the resources, so you kind of know what youre getting into. But is it what happened to our item, a book that wouldve taught a much-needed skill for one of our players?
The log above involves resourceScrapIron, which as you can probably guess is not the book in question. But with that name extracted, we can easily perform an all-time search for the fell off warning message and display all the different items that have been logged:
Continued here:
Splunking the Zombie Apocalypse - Security Boulevard
- The Cranberries' Zombie - the story behind the incendiary song - Louder - April 22nd, 2024
- Chris Riddell on the zombie Tory government falling apart at the seams cartoon - The Guardian - April 22nd, 2024
- Zombie grave dating back 4,200 years discovered in Germany, photos show - Miami Herald - April 22nd, 2024
- Steam zombie survival game leaves early access 11 years on, and you've got one week to get it cheap - PCGamesN - April 22nd, 2024
- "Zombie" cicadas with STDs are coming to Virginia - Axios - April 22nd, 2024
- Deathtroopers is the Star Wars zombie horror game I never knew I needed - PC Gamer - April 22nd, 2024
- Zombie Army VR Reveals The Story In Latest Trailer - Bleeding Cool News - April 22nd, 2024
- Where Infected "Zombie Cicadas" In The US Will Emerge In 2024 - TheTravel - April 22nd, 2024
- Zombie anti-abortion laws are menacing American women - The Hill - April 22nd, 2024
- Joe Manganiello to Zombie It Up in Mountain Man Adaptation - Reactor - Reactor - April 22nd, 2024
- Frozen Soul Covered White Zombie and It Rules - MetalSucks - April 22nd, 2024
- Arizona's abortion ban could bring zombie laws back to life in other states - Fast Company - April 22nd, 2024
- Arizona abortion ban shows the dangers of reviving zombie laws - MSNBC - April 22nd, 2024
- Zombie fires smoldering near Canadian oil and gas wells threaten production following 2023 wildfires - WorldOil - April 22nd, 2024
- You'll Never Guess Who Doesn't Want to Repeal a Zombie Abortion Ban - The New Republic - April 22nd, 2024
- Zombie Army VR: New trailer gives a glimpse of the gruesome campaign - MIXED Reality News - April 22nd, 2024
- Zombie Tests: Is the SAT Back From the Dead? - Fair Observer - April 22nd, 2024
- How likely are you to survive a zombie apocalypse in Connecticut? You may want to travel north - Hartford Courant - April 22nd, 2024
- Missouri Reports 162 New Cases Of Deer 'Zombie' Disease In 2023 - Lake Expo - April 22nd, 2024
- Zombie Fires Pose Looming Threat to Canada's Oil and Gas Infrastructure - OilPrice.com - April 22nd, 2024
- Rob Zombie Presenting House on Haunted Hill Soundtrack on Vinyl - ComicBook.com - April 22nd, 2024
- Take on Zombie Stormtroopers In This Star Wars Fan Game - IGN - April 22nd, 2024
- The Best Zombie Game of 2023 Has a New Expansion Out Now - Esports.net News - April 22nd, 2024
- Frozen Soul Debuts Frosty Rendition of White Zombie's 'Creature of the - Knotfest - April 22nd, 2024
- 'Dawn of the Dead' at 45: A Zombie Love Affair That Never Died - The New York Times - April 12th, 2024
- Hyper-sexual 'zombie cicadas' infected with bizarre STD fungus will emerge in US - New York Post - April 12th, 2024
- Zombie malls and other retail centers getting extreme makeovers to keep up with the times - Chicago Tribune - April 12th, 2024
- 'Zombie' drug xylazine found in cannabis THC vapes in UK - BBC.com - April 12th, 2024
- Taxing health insurance: The Republican zombie that refuses to die - Roll Call - April 12th, 2024
- Fungus-infected Zombie Cicadas with hypersexuality to emerge in the US this year: Expert - Hindustan Times - April 12th, 2024
- Zombie Classic Dawn Of The Dead Coming Back to Theaters for 45th Anniversary - MetalSucks - April 12th, 2024
- E-News | 'Zombie cicadas' infected with sexually transmitted fungus expected to emerge this year in parts of US - WVU ENews - April 12th, 2024
- Deadly Zombie Sedative Penetrating The U.K.s Illicit Drugs Market - Forbes - April 12th, 2024
- Seattle Police warn that base ingredient in 'zombie drug' tranq is being sold as standalone pill - FOX 13 Seattle - April 12th, 2024
- Opinion | Today's Opinions: Haunted by zombie law and thrown into abortion time-travel - The Washington Post - The Washington Post - April 12th, 2024
- Rob Zombie Teases 'House of 1000 Corpses' Book With Rare Photos, Original Script and More! - Bloody Disgusting - April 12th, 2024
- Zombie malls getting extreme makeovers to keep up with the times - Detroit News - April 12th, 2024
- 2000s Horror Fantasy Is One Of The Most Unique Takes On The Zombie Genre - Giant Freakin Robot - April 12th, 2024
- Mysterious 'Zombie Neurons' Unlock Secrets of Learning in The Brain - ScienceAlert - April 12th, 2024
- Zombie SKUs: What They Are, How They Work & Results - JumpFly PPC Advertising News - April 12th, 2024
- Arizona Reviving Its Zombie Abortion Ban Is Trump's Doing - The Cut - April 12th, 2024
- Hyper-sexual "zombie cicadas" that are infected with sexually transmitted fungus expected to emerge this year - CBS News - April 12th, 2024
- Should we be concerned about zombie cicadas? - WGN Radio - Chicago - April 12th, 2024
- You need to play this underrated zombie shooter before it leaves Xbox Game Pass - Digital Trends - April 12th, 2024
- End the 'zombie' Comstock Act - Star Tribune - April 12th, 2024
- OPINION - Flesh-eating zombie narcotics are on our streets Britain is badly losing its war on drugs - Yahoo News UK - April 12th, 2024
- What Is Xylazine - The Flesh-Eating 'Zombie' Drug Linked To 11 Deaths in The UK? - Times Now - April 12th, 2024
- Zombie Cicadas To Emerge In Virginia: What To Know - Patch - April 12th, 2024
- What Is "Zombie Drug" Gripping The UK? - NDTV - April 12th, 2024
- OJ Simpson's Final Film Will Resurrect His Naked Gun Character - As A Zombie? - Looper - April 12th, 2024
- Flesh-eating 'zombie drug' sweeping US is linked to 11 UK deaths - The Telegraph - April 12th, 2024
- Flesh-rotting 'Zombie drug' tranq raises alarm in UK - The Times of India - April 12th, 2024
- Zombie Cicadas To Emerge In GA: What To Know - Patch - April 12th, 2024
- What was the first-ever zombie movie? - Far Out Magazine - April 12th, 2024
- Even Will Smith Couldnt Save Zombie Survival Game Undawn, Which Flopped Spectacularly - IGN - March 24th, 2024
- Rob Zombie Recommends This Surprisingly Wholesome Comedy - Dread Central - March 24th, 2024
- Army Writing and the Zombie (Noun) Apocalypse - Modern War Institute - United States Military Academy West Point - March 24th, 2024
- Fatal Fury's Mai Shiranui Is In This Zombie Survival Game For Some Reason - Time Extension - March 24th, 2024
- Should you be worried about 'zombie deer'? | College of Public Health - George Mason University - March 24th, 2024
- 'Scooby-Doo on Zombie Island' Scares Its Way to Blu-ray for the First Time This Spring - Collider - March 24th, 2024
- Rob Zombie hates my guts: GWAR's Dave 'Oderus Urungus' Brockie was a genius, and here's an interview that proves it - Louder - March 24th, 2024
- In the fight against gun violence, Rep. Frost Introduces the Destroy Zombie Guns Act - The Apopka Voice - March 24th, 2024
- Elfriede Jelinek's 'The Children of the Dead' is a historical zombie novel - The Washington Post - The Washington Post - March 24th, 2024
- Germany's Zombie Government Is Fueling the Far Right - The Atlantic - March 24th, 2024
- Zombie Survival Game Undawn Has Crashed And Burned According To A Reuters Report - MMOs.com - March 24th, 2024
- Will Smiths Zombie Game Has Been Hit Harder Than Chris Rock at the Oscars Its Astronomical Budget Was ... - imdb - March 24th, 2024
- Scooby-Doo On Zombie Island / Return to Zombie Island Blu-ray (Warner Archive Collection) - Blu-ray.com - March 24th, 2024
- Registration almost here for Dothans annual Zombie 5K Chase - WDHN - March 24th, 2024
- Zombie magic and gambling club Magic Island will come alive again this summer - Houston Chronicle - March 24th, 2024
- Undawn | The $140m zombie survival game starring Will Smith that bombed while nobody was looking - Filmstories - March 24th, 2024
- After 4 years, a cult classic zombie horror manga from Ga-Rei's creator is finally getting an official English release - Gamesradar - March 24th, 2024
- Will Smith's Zombie Game Has Been Hit Harder Than Chris Rock at the Oscars - Its Astronomical Budget Was ... - FandomWire - March 24th, 2024
- "The game literally no one knew existed": The Will Smith Zombie Survival Game is Reportedly a Big Flop and Fans Are ... - FandomWire - March 24th, 2024
- Zombie-palooza --- Get to the biggest undead music event in the latest Dead Island 2 expansion: SoLA - Gaming Trend - March 24th, 2024
- Bournemouth University medical students treat 'zombie attack' in mass training simulation - The Tab - March 24th, 2024
- Zombie car factories on the rise in China as buyers opt for EVs - Financial Times - March 16th, 2024
- Hey, Remember When Rob Zombie Was Going To Direct THE CROW 3? - FANGORIA - March 16th, 2024
- New Walking Dead Twist Shows How The Zombie Outbreak Will Finally End - Screen Rant - March 16th, 2024
- Candid conversations from death row with The Zombie Hunter - Arizona's Family - March 16th, 2024
- Zombie! The Musical - REVIEW - City Hub Sydney - March 16th, 2024
Reviewed and Recommended by Erik Baquero