Console Producer and Consumer Basics

Question:

What is the simplest way to write messages to and read messages from Kafka?

Edit this page

Example use case:

So you are excited to get started with Kafka and you'd like to produce and consume some basic messages and you want to do so quickly. In this tutorial we'll show you how to produce and consume messages from the command line with no code!


TweetShare your tutorial progress

Hands-on code example:

New to Confluent Cloud? Sign up and then run this tutorial.




Short Answer

With Confluent Cloud, you can use the Confluent Cloud CLI to produce and consume messages.

Producer:

ccloud kafka topic produce orders --parse-key --delimiter ":"

Consumer:

ccloud kafka topic consume orders --print-key --delimiter "-" --from-beginning

Run it

1
Provision your Kafka cluster

The quickest way to get started with Apache Kafka is on Confluent Cloud, which provides it as a fully managed service. First, sign up for Confluent Cloud.

  1. After you log in to Confluent Cloud, click on Add cloud environment and name the environment learn-kafka. Using a new environment keeps your learning resources separate from your other Confluent Cloud resources.

  2. From the Billing & payment section in the Menu, apply the promo code CC100KTS to receive an additional $100 free usage on Confluent Cloud (details).

  3. Click on LEARN and follow the instructions to launch a Kafka cluster and to enable Schema Registry.

Confluent Cloud

2
Download and setup the Confluent Cloud CLI

This tutorial has some steps for Kafka topic management and/or reading from or writing to Kafka topics, for which you can use the Confluent Cloud Console or install the Confluent Cloud CLI. Instructions for installing Confluent Cloud CLI and configuring it to your Confluent Cloud environment is available from within the Confluent Cloud Console: navigate to your Kafka cluster, click on the CLI and tools section, and run through the steps in the CCloud CLI tab.

3
Create the Kafka topic

Create a Kafka topic called orders in Confluent Cloud.

ccloud kafka topic create orders --partitions 1

This should yield the following output:

Created topic "orders".

4
Start a console consumer

Next, let’s open up a consumer to read records.

ccloud kafka topic consume orders

5
Produce events to the Kafka topic

Now we are going to produce records to our new topic.

ccloud kafka topic produce orders

Then enter these records either one at time or copy-paste all of them into the terminal and hit enter:

{"number":1,"date":18500,"shipping_address":"ABC Sesame Street,Wichita, KS. 12345","subtotal":110.00,"tax":10.00,"grand_total":120.00,"shipping_cost":0.00}
{"number":2,"date":18501,"shipping_address":"123 Cross Street,Irving, CA. 12345","subtotal":5.00,"tax":0.53,"grand_total":6.53,"shipping_cost":1.00}
{"number":3,"date":18502,"shipping_address":"5014  Pinnickinick Street, Portland, WA. 97205","subtotal":93.45,"tax":9.34,"grand_total":102.79,"shipping_cost":0.00}
{"number":4,"date":18503,"shipping_address":"4082 Elmwood Avenue, Tempe, AX. 85281","subtotal":50.00,"tax":1.00,"grand_total":51.00,"shipping_cost":0.00}
{"number":5,"date":18504,"shipping_address":"123 Cross Street,Irving, CA. 12345","subtotal":33.00,"tax":3.33,"grand_total":38.33,"shipping_cost":2.00}

6
Produce records with full key-value pairs

Kafka works with key-value pairs, but so far you’ve only sent records with values only. Well to be fair you’ve sent key-value pairs, but the keys are null. Sometimes you’ll need to send a valid key in addition to the value from the command line.

To enable sending full key-value pairs from the command line you add two properties to your Confluent Cloud CLI, parse-key and delimiter

Let’s try to send some full key-value records now. If your previous console producer is still running close it with a CTRL+C and run the following command to start a new console producer:

ccloud kafka topic produce orders --parse-key --delimiter ":"

Then enter these records either one at time or copy-paste all of them into the terminal and hit enter. In the output below, the 6 and 7 are the keys, separated from the values by the : delimiter.

6:{"number":6,"date":18505,"shipping_address":"9182 Shipyard Drive, Raleigh, NC. 27609","subtotal":72.00,"tax":3.00,"grand_total":75.00,"shipping_cost":0.00}
7:{"number":7,"date":18506,"shipping_address":"644 Lagon Street, Chicago, IL. 07712","subtotal":11.00,"tax":1.00,"grand_total":14.00,"shipping_cost":2.00}

7
Start a consumer to show full key-value pairs

Next, let’s open up a consumer to read the new records.

ccloud kafka topic consume orders --print-key --delimiter "-" --from-beginning

After the consumer starts you should see the following output in a few seconds:

null-{"number":1,"date":18500,"shipping_address":"ABC Sesame Street,Wichita, KS. 12345","subtotal":110.00,"tax":10.00,"grand_total":120.00,"shipping_cost":0.00}
null-{"number":2,"date":18501,"shipping_address":"123 Cross Street,Irving, CA. 12345","subtotal":5.00,"tax":0.53,"grand_total":6.53,"shipping_cost":1.00}
null-{"number":3,"date":18502,"shipping_address":"5014  Pinnickinick Street, Portland, WA. 97205","subtotal":93.45,"tax":9.34,"grand_total":102.79,"shipping_cost":0.00}
null-{"number":4,"date":18503,"shipping_address":"4082 Elmwood Avenue, Tempe, AX. 85281","subtotal":50.00,"tax":1.00,"grand_total":51.00,"shipping_cost":0.00}
null-{"number":5,"date":18504,"shipping_address":"123 Cross Street,Irving, CA. 12345","subtotal":33.00,"tax":3.33,"grand_total":38.33,"shipping_cost":2.00}
6-{"number":6,"date":18505,"shipping_address":"9182 Shipyard Drive, Raleigh, NC. 27609","subtotal":72.00,"tax":3.00,"grand_total":75.00,"shipping_cost":0.00}
7-{"number":7,"date":18506,"shipping_address":"644 Lagon Street, Chicago, IL. 07712","subtotal":11.00,"tax":1.00,"grand_total":14.00,"shipping_cost":2.00}

Since we kept the --from-beginning property, you’ll see all the records sent to the topic. You’ll notice the results before you sent keys are formatted as null-<value>.

8
Teardown Confluent Cloud resources

You may try another Kafka tutorial, but if you don’t plan on doing other tutorials, use the Confluent Cloud Console or CLI to destroy all the resources you created. Verify they are destroyed to avoid unexpected charges.