Type Safe JSON Parsing in Go

Code Examples

There are a number of code snippets throughout this post to illustrate different approaches. If you want to see the full code for any of these examples, you can check out the repo for the tech talk this blog was based on. These examples deal with a fictional JSON API for describing payments. The assumption is that to process a payment we’re going to need to parse and understand information about a number of individual payments. In place of payment execution logic we’re just printing the details of each payment we get. We’re going to leave out the details of error checking, instead “handling” any errors with panic.

The Basics

This post assumes that you’re mostly familiar with the standard library’s encoding/json package, but we have a simple example in case you need a refresher. Our JSON data here is simple. We have an array of objects that each represent a single payment. Each payment has an id, an amount, a to, and a from field. Each of those fields values is a string.

{
    "payments": [
        {
            "id": "12345",
            "amount": "3.50",
            "to": "nessie",
            "from": "packrat386"
        },
        {
            "id": "12346",
            "amount": "0.02",
            "to": "packrat386",
            "from": "mlarraz"
        }
    ]
}

To parse this data we would create two matching structs. One struct, PaymentCollection, represents the top level array of payments. The other struct, Payment, represents an individual payment. We tag each of the struct fields with the corresponding field in the JSON data and then we can parse into an instance of PaymentCollection using json.Unmarshal.

type PaymentCollection struct {
  Payments []Payment `json:"payments"`
}

type Payment struct {
  ID     string `json:"id"`
  Amount string `json:"amount"`
  To     string `json:"to"`
  From   string `json:"from"`
}

func main() {
  pmts := new(PaymentCollection)
  err := json.Unmarshal([]byte(data), pmts)
  if err != nil {
    panic(err)
  }

  fmt.Printf("%#v\n", pmts)
}

The standard library parser will ensure that the struct fields will be populated by the matching JSON fields (as long as the fields exist in the JSON data). It’s easy to use and ensures that the types in the data line up with the types we’ve declared in the struct.

Motivating Example

So what kind of JSON structure makes it difficult to use simple struct tags? Generally the issue arises from fields that could have more than one type. In our payment API example we could imagine that each payment contains a type field and a payment_details field. The contents of payment_details will differ depending on which type of payment we have. We have ACH payments that require a routing number and an account number and we have credit card payments that require a card number and an expiration date.

{
    "payments": [
        {
            "id": "12345",
            "amount": "3.50",
            "type": "ach",
            "payment_details": {
                "routing_number": "123456789",
                "account_number": "987654321"
            }
        },
        {
            "id": "12345",
            "amount": "3.50",
            "type": "credit_card",
            "payment_details": {
                "card_number": "1111222233334444",
                "expiration": "0618"
            }
        }
    ]
}

We want a way to execute each payment in our collection, and that will require information in payment_details. But how can we make our types in such a way that it can deal with two possibilities for the contents of payment_details?

Union Type

The low tech approach to this is to try to define a type that can contain all the possibilities of the payment_details field. As before, we have a PaymentCollection type and a Payment type that now has a PaymentDetails field. That field is a Details struct that contains fields corresponding to both ACH payment details (routing_number and account_number) and credit card payment details (card_number and expiration). I generally call this kind of struct a “union” type since you can consider it to be the union of two possible types.

type PaymentCollection struct {
  Payments []Payment `json"payments"`
}

type Payment struct {
  ID             string `json:"id"`
  Amount         string `json:"amount"`
  Type           string `json:"type"`
  PaymentDetails Detail `json:"payment_details"`
}

type Detail struct {
  RoutingNumber string `json:"routing_number"`
  AccountNumber string `json:"account_number"`
  CardNumber    string `json:"card_number"`
  Expiration    string `json:"expiration"`
}

Parsing into an instance of PaymentCollection is as easy as calling json.Unmarshal as in the earlier example. We can then implement Execute() for our payment struct. Our code conditionally switches its interpretation of the data contained in Detail based on the type of payment it comes from.

func (p Payment) Execute() {
  if p.Type == "ach" {
    fmt.Printf(
      "Executing ach payment\nrouting_number: %s\naccount_number: %s\n",
      p.PaymentDetails.RoutingNumber,
      p.PaymentDetails.AccountNumber,
    )
  } else if p.Type == "credit_card" {
    fmt.Printf(
      "Executing credit card payment\ncard_number: %s\nexpiration: %s\n",
      p.PaymentDetails.CardNumber,
      p.PaymentDetails.Expiration,
    )
  } else {
    panic("unrecognized type")
  }
}

This can be a viable approach when we only have a relatively small number of cases to handle. We maintain easy compatibility with the parsing patterns of the standard library and we still get our guarantees of type safety. However we can run into issues if we have a large number of possible types. If we were to support ten different payment methods instead of two we would be dealing with a struct that has 20 or 30 fields and only ever populates a small number of them. We also can’t use this approach to parse data where a single field could have two types of values in it, for example an idfield that could be a string or a number. My main objection to using this approach even when it’s technically feasible is that our Detail type can’t be interpreted without the context of knowing what type of payment it came from. If we ever wanted to separate the Detail from the rest of the Payment we would have to make sure to always pass around the type with it, otherwise code dealing with our Detail won’t know which fields will be populated.

map[string]interface{}

Perhaps the most common approach to this problem that I encounter in the wild is to use map[string]interface{} to represent an arbitrary JSON object. We again define a PaymentCollection type and a Payment type, and this time our PaymentDetails field will have the type of map[string]interface{} which can contain both forms of the payment_details field in our JSON.

type PaymentCollection struct {
  Payments []Payment `json"payments"`
}

type Payment struct {
  ID             string                 `json:"id"`
  Amount         string                 `json:"amount"`
  Type           string                 `json:"type"`
  PaymentDetails map[string]interface{} `json:"payment_details"`
}

Once again we can parse into an instance of PaymentCollection using json.Unmarshal. To implement our Execute()method we need look at the type of the payment and then cast the data in the map to the appropriate type.

func (p Payment) Execute() {
  if p.Type == "ach" {
    routingNumber, ok := p.PaymentDetails["routing_number"].(string)
    if !ok {
      panic("routing number not a string")
    }

    accountNumber, ok := p.PaymentDetails["account_number"].(string)
    if !ok {
      panic("account number not a string")
    }

    fmt.Printf(
      "Executing ach payment\nrouting_number: %s\naccount_number: %s\n",
      routingNumber,
      accountNumber,
    )
  } else if p.Type == "credit_card" {
    cardNumber, ok := p.PaymentDetails["card_number"].(string)
    if !ok {
      panic("routing number not a string")
    }

    expiration, ok := p.PaymentDetails["expiration"].(string)
    if !ok {
      panic("account number not a string")
    }

    fmt.Printf(
      "Executing credit card payment\ncard_payment: %s\nexpiration: %s\n",
      cardNumber,
      expiration,
    )
  } else {
    panic("unrecognized type")
  }
}

This implementation has the main benefit of requiring very little up front planning on our part. We don’t have to define any special types and we get to just use json.Unmarshal and figure out what we’re working with later. However, there are significant downsides to using interface{} this way. For starters, we’re giving up on the type checking built in to the standard library parser. If someone sends us a number for card_number instead of a string, we’re not going to find out until we try to cast it in Execute(). This means that we have to add a lot of casting and error checking whenever we want to do anything with the data. This implementation has an even worse issue with required context than the union type example. If our payment detail information gets separated from the Payment struct, there’s no way to even know that it’s meant to be payment detail information, much less which type of payment detail information we should interpret it as. If there’s a single takeaway in my experience working with JSON in Go, it’s that map[string]interface{} is used far too often. Trying to avoid spending time working out the structure of the data usually ends up incurring large maintenance costs.

JSON Structure

To properly deal with complicated JSON data, we need to understand the structure of the JSON data format as a whole. JSON is actually quite simple in its definition. A properly formatted JSON document will be one of two types at the top level.

The first type is an array, which is an ordered list of values.

The other possibility for a top level type is an object, which is a mapping from keys (strings) to values.

Both of these top level types are containers that contain values. A value is one object, array, string, number, boolean, or null value. This means that all JSON data is a value (which may contain other values).

Many scripting languages have containers that can contain many different kinds of values, like Python’s dict or Ruby’s Hash. In languages with this type of container, parsing JSON is trivial since we can map each possibility for value to primitive types in the language and then just parse into instances of those types without having to declare anything special about the structure.

The issue in Go is that our containers have to specify a single type that they contain, and outside of interface{} we don’t have a single type that can represent all the possible things a value can be, and it’d be difficult to define such a type since there’s no real overlap between all those things. If we don’t specify a structure our only option for parsing arbitrary data is to use interface{}, which comes with all the issues we just discussed.

RawMessage

So the core of the problem is that we have some field whose data we can’t really parse without some other information. What we need is a way to put off parsing that field until we have the information we need to successfully parse it. Luckily for us, the standard library gives us some help here with the type json.RawMessage. json.RawMessage is actually a very simple type. It’s just a type alias for []byte that also defines MarshalJSON and UnmarshalJSON as no-ops. This means it fulfills the json.Marshaler interface, but any data we put into it or read out of it won’t actually be parsed into any concrete type. These properties mean that we can use it as a temporary stand-in for any value that want to wait to parse later.

Implementing our payment types with json.RawMessage isn’t particularly difficult. We once again define PaymentCollection and Payment types, and this time the Detail field of a Payment is of the type json.RawMessage.

type PaymentCollection struct {
  Payments []*Payment `json"payments"`
}

type Payment struct {
  ID     string          `json:"id"`
  Amount string          `json:"amount"`
  Type   string          `json:"type"`
  Detail json.RawMessage `json:"payment_details"`
}

In addition to those types, we’re also going to define structs for the two different kinds of payment details. achDetail represents the payment details of an ACH payment, and cardDetail represents the details of a credit card payment. Each of those types has the struct tags required to parse JSON into them and defines Execute() to print the relevant information.

type achDetail struct {
  RoutingNumber string `json:"routing_number"`
  AccountNumber string `json:"account_number"`
}

func (a *achDetail) Execute() {
  fmt.Printf(
    "Executing ach payment\nrouting_number: %s\naccount_number: %s\n",
    a.RoutingNumber,
    a.AccountNumber,
  )
}

type cardDetail struct {
  CardNumber string `json:"card_number"`
  Expiration string `json:"expiration"`
}

func (c *cardDetail) Execute() {
  fmt.Printf(
    "Executing cc payment\ncard_number: %s\nexpiration: %s\n",
    c.CardNumber,
    c.Expiration,
  )
}

As in the other examples we can json.Unmarshal into an instance of PaymentCollection. For our implementation of Execute(), we’re going to use the information from the type field that we’ve already parsed to correctly parse the contents of Details (which is just the JSON data as a RawMessage) into one of the two concrete types (achDetail or cardDetail). Then we can just call the Execute() method that both of those types expose.

type Executor interface {
  Execute()
}

func (p Payment) Execute() {
  var internal Executor
  if p.Type == "ach" {
    internal = new(achDetail)
  } else if p.Type == "credit_card" {
    internal = new(cardDetail)
  } else {
    panic("unrecognized type")
  }

  err := json.Unmarshal(p.Detail, internal)
  if err != nil {
    panic(err)
  }
  internal.Execute()
}

This approach gives us two real types to represent the two possibilities for payment detail information. We get the guarantees of type safety that the standard library parser provides without having to do any casting. Having real types also addresses the context issue that we had with our earlier solutions. We don’t need any outside information to use an instance of cardDetail or achDetail because they can only be one thing. We’re able once again to use the language’s type system to describe the type of data we’re dealing with, which is certainly preferable to passing around our type as a string!

The only cost to this approach is in planning. We need to think through the possible forms of the data, define types for each form, and parse our data in multiple steps where we intelligently pick a type to unmarshal into once we have the information we need. There are ways to make this smoother (which we will discuss below), but at the end of the day this is the work that a language like Go requires. If we want to use the type system to our benefit then we have to think about and define types for the data we’re working with, and there’s no easy replacement for doing that work by hand.

Beautifying RawMessage

What if we wanted to make the more complicated multi-step parsing process seem like it’s all happening in a single step? We can use the magic of Embedded Types to make it appear as if we still only have a single Payment type. A caveat to this section: embedded types are complicated enough to warrant their own long blog post, so I won’t explain all their intricacies here.

This time we’re going to define Payment in terms of two embedded types. We first embed a paymentData struct that defines the fields we’re working with and struct tags to parse JSON into them. Our paymentData struct defines its Detail field as a json.RawMessage. We then embed into Payment a Detail interface that defines the Execute()method. This means our Payment will “feel” like the Payment we had before since it will have several concrete fields and an Execute() method that uses data from payment_details.

type Payment struct {
  paymentData
  Detail
}

type paymentData struct {
  ID     string          `json:"id"`
  Amount string          `json:"amount"`
  Type   string          `json:"type"`
  Detail json.RawMessage `json:"payment_details"`
}

type Detail interface {
  Execute()
}

The next step is to define our two concrete detail types that implement the Detail interface. As before, we have one for ACH (achDetail) and one for credit cards (cardDetail).

type achDetail struct {
  RoutingNumber string `json:"routing_number"`
  AccountNumber string `json:"account_number"`
}

func (a *achDetail) Execute() {
  fmt.Printf(
    "Executing ach payment\nrouting_number: %s\naccount_number: %s\n",
    a.RoutingNumber,
    a.AccountNumber,
  )
}

type cardDetail struct {
  CardNumber string `json:"card_number"`
  Expiration string `json:"expiration"`
}

func (c *cardDetail) Execute() {
  fmt.Printf(
    "Executing cc payment\ncard_number: %s\nexpiration: %s\n",
    c.CardNumber,
    c.Expiration,
  )
}

The way we make the JSON parsing look seamless is to implement the json.Marshaler interface. For a given type (Payment in this case) we can give the standard library parser specific instructions for how to unmarshal data into and marshal data from instances of that type.

First we define UnmarshalJSON. We first unmarshal the data into our embedded paymentData struct. This will populate the static fields (ID, Amount, and Type) and store the contents of payment_details as a json.RawMessage. We then unmarshal the contents of payment_details into a concrete type that we pick based on the type of the payment, and assign that type to our embedded Detail interface.

func (p *Payment) UnmarshalJSON(data []byte) error {
  err := json.Unmarshal(data, &p.paymentData)
  if err != nil {
    return err
  }

  var detail Detail
  if p.Type == "ach" {
    detail = new(achDetail)
  } else if p.Type == "credit_card" {
    detail = new(cardDetail)
  } else {
    return fmt.Errorf("unrecognized type")
  }

  err = json.Unmarshal(p.paymentData.Detail, detail)
  if err != nil {
    return err
  }
  p.Detail = detail
  return nil
}

For the sake of completeness we can also define MarshalJSON. We first marshal our embedded Detail into paymentData‘s Detail field, and then we marshal the embedded paymentData to JSON and return that.

func (p *Payment) MarshalJSON() ([]byte, error) {
  data, err := json.Marshal(p.Detail)
  if err != nil {
    return make([]byte, 0), err
  }

  p.paymentData.Detail = data

  return json.Marshal(p.paymentData)
}

Now when we go to write our main function everything seems to work smoothly. We can unmarshal into a PaymentCollection, loop over and Execute() the resulting payments, and marshal the collection back to JSON if we want to. The complexity of the two-step parsing process is abstracted by the json.Marshaler interface.

func main() {
  pmts := new(PaymentCollection)
  err := json.Unmarshal([]byte(data), pmts)
  if err != nil {
    panic(err)
  }

  for _, p := range pmts.Payments {
    p.Execute()
  }

  output, err := json.Marshal(pmts)
  if err != nil {
    panic(err)
  }

  fmt.Println(string(output))
}

By using json.RawMessage and parsing incrementally we can make our code just as idiomatic and easy to use as in the simple case from the beginning of the post. We can do that without having to compromise on safe and meaningful types because json.RawMessage gives us a good type to represent data that we haven’t gotten around to parsing yet. Using this approach requires an up front investment in understanding the structure of your data, but I think in the long run that investment will pay off in terms of ease of understanding and maintaining code dealing with JSON data.

Citations

• The standard library package encoding/json is what makes all of this possible. It’s well worth reading through and understanding some of the more niche utilities the library offers.
• json.org is an excellent resource on the general structure and specification of JSON. The images describing the structure of a JSON document are taken from there.
• Effective Go is a a great style guide and explanation for much of the language, and in particular was referenced to explain embedded types.