Parsing Log Files into JSON for Elasticsearch and Kibana

Three AM. PagerDuty goes off. Production is throwing 500 errors and nobody knows why.

You SSH into the box, tail the logs, and stare at thousands of lines that look like this:

2024-01-15 03:02:44 ERROR [RequestHandler] Connection timeout after 30000ms - user_id=48271 endpoint=/api/checkout retry_count=3

Useful information is buried in there. Timestamps, error types, user IDs, endpoints. But grep can only get you so far. You need to query this data. Filter by endpoint. Count errors per user. Graph timeout frequency over time.

That means getting these logs into Elasticsearch. And Elasticsearch wants JSON.

Why Logs Need Structure

Raw log lines are strings. Just text. Elasticsearch can store them, but you can’t do much beyond full-text search. Want to find all errors from the last hour? You’re searching for the word “ERROR” and hoping your timestamp format cooperates.

JSON gives you fields. When your log becomes:

{
  "timestamp": "2024-01-15T03:02:44",
  "level": "ERROR",
  "component": "RequestHandler",
  "message": "Connection timeout after 30000ms",
  "user_id": 48271,
  "endpoint": "/api/checkout",
  "retry_count": 3
}

Now you can query properly. Give me all ERROR level entries where endpoint equals /api/checkout and retry_count is greater than 2. Elasticsearch handles that in milliseconds across millions of documents.

Kibana dashboards become possible. Time-series graphs of error rates. Pie charts breaking down errors by component. Tables showing which users hit the most failures. None of that works with unstructured text.

The Log Parsing Problem

Converting logs to JSON sounds simple until you actually try it. Logs are inconsistent. Different services format differently. Even the same service changes format between versions.

Here’s what I pulled from three different services in one infrastructure:

Apache:

192.168.1.105 - - [15/Jan/2024:03:02:44 +0000] "GET /images/logo.png HTTP/1.1" 200 2048

Application:

2024-01-15 03:02:44 ERROR [RequestHandler] Connection timeout - user_id=48271

Nginx:

2024/01/15 03:02:44 [error] 1234#0: *5678 upstream timed out (110: Connection timed out)

Three completely different formats. Three different date patterns. Three different ways of indicating severity. You can’t write one regex to rule them all.

Production environments have dozens of log formats. Load balancers, application servers, databases, caches, message queues—each has opinions about how logs should look.

Logstash Grok Patterns

The traditional answer is Logstash with Grok patterns. Grok is essentially named regex. Instead of writing (d{4}-d{2}-d{2}) you write %{TIMESTAMP_ISO8601:timestamp}.

A basic Logstash config looks like:

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} [%{DATA:component}] %{GREEDYDATA:message}" }
  }
}

Grok works. It’s battle-tested. Half the internet runs on Logstash pipelines parsing logs this way.

But Grok configs get ugly fast. Complex log formats need patterns nested three layers deep. Conditional logic for multiline exceptions. Error handling for malformed entries. A production Logstash config for a real application easily runs hundreds of lines.

And when your pattern doesn’t match? The log line hits Elasticsearch as unparsed garbage, or worse, gets dropped entirely. Debugging Grok patterns at 3 AM when you’re trying to figure out why production is down is not fun.

When You Just Need Quick Conversion

Sometimes you don’t need a production pipeline. You have a log file. You need it in JSON. Once.

Maybe you’re debugging locally and want to load logs into a JSON viewer. Maybe you’re building a proof of concept before committing to a full ELK deployment. Maybe you grabbed logs from a server that’s about to be decommissioned and just need to archive them in a queryable format.

For one-off conversions, spinning up Logstash is overkill. You need something that can take text and output JSON without configuring a pipeline.

Tools that convert plain text into JSON handle the structural transformation without the ceremony. Paste your log lines, define how fields should split, get JSON back. No config files, no pipeline setup, no Java heap tuning.

I use this approach during incident investigation when I need to quickly analyze a log excerpt someone posted in Slack. Copy the text, convert to JSON, throw it into a local query tool. Faster than getting access to the actual logging infrastructure.

Structuring Unstructured Data

The conversion process forces you to think about structure. What parts of this log line are distinct fields? Where does the timestamp end and the message begin? Which pieces need to be queryable versus just stored?

Take this line:

2024-01-15 03:02:44 ERROR [RequestHandler] Connection timeout after 30000ms - user_id=48271 endpoint=/api/checkout retry_count=3

Obvious fields:

• Timestamp: 2024-01-15 03:02:44
• Level: ERROR
• Component: RequestHandler

Less obvious:

• Is “Connection timeout after 30000ms” one message field, or should timeout duration be extracted separately?
• The key-value pairs at the end (user_id, endpoint, retry_count)—flatten them into top-level fields or nest them under a “context” object?

There’s no universally right answer. It depends on how you’ll query the data. If you need to filter by timeout duration, extract it. If you just need to read the message, keep it together.

The best structure mirrors your questions. What will you ask this data? Structure it so those queries are simple.

Handling Key-Value Pairs

Many modern log formats include structured data within the unstructured line. That user_id=48271 endpoint=/api/checkout pattern is everywhere.

Some converters recognize this automatically. They see key=value patterns and expand them into proper JSON fields. Others need you to specify the pattern or delimiter.

The tricky part is mixed content. A log line might have:

Processing request for user_id=48271 to endpoint=/api/checkout

The sentence “Processing request for” contains “for” which looks like it could be a key. Naive key-value parsing might mangle this. Smarter parsing looks for the = sign and works backward to find the key.

When automatic parsing fails, manual field definition saves the day. You tell the converter: characters 0-19 are timestamp, 20-25 are level, and so on. Tedious, but reliable.

Timestamps Deserve Special Attention

Elasticsearch needs ISO 8601 timestamps to enable time-based queries. Your logs probably don’t use ISO 8601.

Common formats I’ve seen:

• 2024-01-15 03:02:44 (close but missing the T)
• 15/Jan/2024:03:02:44 +0000 (Apache common log format)
• Jan 15 03:02:44 (syslog, no year)
• 1705287764 (Unix epoch seconds)
• 1705287764000 (Unix epoch milliseconds)

Converting these correctly matters. Get the timestamp wrong and your time-series graphs are useless. Kibana will show all your logs at the wrong time or refuse to parse them at all.

The safest approach: normalize everything to ISO 8601 during conversion. 2024-01-15T03:02:44.000Z. Elasticsearch understands it natively. Kibana renders it correctly. No ambiguity.

Multiline Log Entries

Stack traces break everything.

A Java exception might span fifty lines. One error event, fifty log lines. If you parse line-by-line, you get fifty JSON documents when you wanted one.

2024-01-15 03:02:44 ERROR [RequestHandler] Unhandled exception
java.lang.NullPointerException: Cannot invoke method on null object
    at com.example.RequestHandler.process(RequestHandler.java:142)
    at com.example.RequestHandler.handle(RequestHandler.java:89)
    at com.example.Server.dispatch(Server.java:201)

Logstash has multiline codecs that accumulate lines until they see the next timestamp. That works for streaming ingestion.

For batch conversion, you need to pre-process. Identify where entries begin (lines starting with timestamps) and concatenate continuation lines into single entries before converting. Some tools handle this automatically if you specify a start pattern.

Or store the stack trace as a single string field with embedded newlines. Less elegant, but the query “show me all NullPointerExceptions” still works.

Testing Your JSON Output

Before bulk-loading converted logs into Elasticsearch, validate a sample.

Paste a few JSON objects into a validator. Are they syntactically correct? Missing commas and unclosed brackets are easy mistakes.

Check field types. Did numeric values stay numeric, or did they become strings? Elasticsearch behaves differently for {"count": 5} versus {"count": "5"}. You can’t do math on string numbers.

Verify timestamps. Load one document into Elasticsearch and check that the @timestamp field shows the right time in Kibana. Timezone issues hide here—a log from 3 AM might display as 11 PM if you converted without timezone awareness.

Spot-check a few specific entries against the original logs. Does the JSON contain all the information from the original line? Did any fields get truncated or mangled?

Production Pipeline vs. One-Off Analysis

The right tool depends on your workflow.

Building dashboards that update in real-time as logs arrive? You need a proper pipeline—Logstash, Fluentd, Vector, or whatever your infrastructure already runs. Configuration overhead is justified because it runs continuously.

Investigating a specific incident with a finite set of logs? One-off conversion gets you to answers faster. Convert the relevant files, load them somewhere queryable, find the problem, move on.

I keep both approaches ready. The production pipeline handles ongoing monitoring. Quick converters handle ad-hoc investigation. Different problems, different tools.

What Structured Logs Enable

Once your logs are JSON in Elasticsearch, doors open.

Correlation across services. When user 48271 complains about a failed checkout, you can query every service—API gateway, payment processor, inventory system—for logs mentioning that user ID. The error chain becomes visible.

Anomaly detection. Establish baseline error rates, then alert when errors spike. Hard to do with grep, trivial with structured queries.

Long-term trending. Is the checkout timeout problem getting worse? Query last month’s logs and graph the frequency over time. Patterns emerge that you’d never notice reading individual log lines.

Accountability. When the post-mortem asks “how long was the system degraded before we noticed?” you can answer precisely. The data exists, structured and queryable, waiting for the right question.

All because you converted text to JSON.