vlambda博客
学习文章列表

Concepts

https://docs.confluent.io/current/streams/concepts.html

Source connector- fetch from data source

Sink connector- export to data destination

Get hands duty 

Docker image for confluent platform

https://docs.confluent.io/current/quickstart/ce-docker-quickstart.html

Docker configuration parameters

https://docs.confluent.io/current/installation/docker/config-reference.html

Delete a topic

$ docker-compose -p confluent exec zookeeper kafka-topics --topic <topicname> --zookeeper zookeeper --delete

Secure Kafka

https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/

Clients

Connect a custom application written other than Java

https://www.confluent.io/clients/

https://docs.confluent.io/current/clients/index.html#kafka-clients

Apache Avro (Schema definition)

https://www.confluent.io/blog/avro-kafka-data/

Avro is an open source data serialization system that helps with data exchange between systems, programming languages, and processing frameworks. Avro helps define a binary format for your data, as well as map it to the programming language of your choice.

Kafka Connect JDBC Source Connector

https://docs.confluent.io/current/connect/kafka-connect-jdbc/index.html

Kafka Connect Deep Dive – JDBC Source Connector

Bytes, Decimals, Numerics and oh my

https://www.confluent.io/blog/kafka-connect-deep-dive-jdbc-source-connector

Configs options

https://docs.confluent.io/current/connect/kafka-connect-jdbc/source-connector/source_config_options.html

MSSQL 

The JDBC source and sink connectors include the open source jTDS JDBC driver to read from and write to Microsoft SQL Server. Because the JDBC 4.0 driver is included, no additional steps are necessary before running a connector to Microsoft SQL Server.

Alternatively, you can remove the jTDS JDBC driver and install the open source Microsoft JDBC driver. First, download the latest version of the JDBC driver archive (for example, sqljdbc_7.2.2.0_enu.tar.gz for English), extract the contents of the file to a temporary directory, and find the correct JAR file for your version of Java. For example, if downloading the 7.2.2.0 version of the driver, find either the mssql-jdbc-7.2.2.jre8.jar if running Connect on Java 8 or the mssql-jdbc-7.2.2.jre11.jar if running Connect on Java 11.

变更数据捕获（Change Data Capture / CDC）

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.SQLServer.CommonDBATasks.CDC.html

https://debezium.io/docs/connectors/sqlserver/

Troubleshooting

KSQL select has no results

https://docs.confluent.io/current/ksql/docs/troubleshoot-ksql.html

$ docker network ls# Example outputNETWORK ID NAME DRIVER SCOPE287f2fc55e93 bridge bridge local885cbacd0693 confluent_default bridge local

# Use kafkacat to consume the message and print the summary$ docker run --network confluent_default --tty --interactive --rm \ confluentinc/cp-kafkacat \ kafkacat -b kafka:39092 \ -C -t pageviews \. # change pageviews to the topic name -o beginning# Example output% ERROR: Failed to query metadata for topic ERP_SALES_ORDERS: Local: Broker transport failure