Can't create custom Kafka Connector
I am running a centOS docker container in an ec2 instance. I've installed Confluent, the necessary java files and placed them in the right places, and Kafka/ZooKeeper etc. are all up and running.
My goal is to publish from a MySQL db to Kafka. I've installed the mysql-connector jdbc (mysql-connector-java.noarch) and the jar file (mysql-connector-java.jar) is in the folder with the kafka connect jar file. I also installed jq, because Confluent kept badgering me to.
I am trying to create a custom connector to read from a remote database. Using the provided SQLite connector as a model (/etc/kafka-connect-jdbc/source-quickstart-sqlite.properties) I attempted to create my own connector file for my mysql database:
#mysql kafka publishing test
#
name=mysql-bulk-source
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=10
connection.url=jdbc:mysql://localhost:3306
mode=bulk
connection.user=kafkadbtest
connection.password=password
topic.prefix=mysql-
but when I run confluent load mysql-source -d /path/to/myfile/mysql-source.properties I get no output, and confluent status connectors is empty. Running confluent config mysql-connector -d /path/mysql-source.properties also does nothing, and following up with confluent config mysql-connector returns an empty config.
Confluent connect is up. I don't have any other ideas of what to try. Anyone can help? Any more info on my config that I should provide?
(p.s. there is no kafka tag, but it seems like there should be, and I lack the reputation to create it.)
centos mysql docker
asked Feb 6 at 21:06
Y e zY e z
1046
add a comment |
I am running a centOS docker container in an ec2 instance. I've installed Confluent, the necessary java files and placed them in the right places, and Kafka/ZooKeeper etc. are all up and running.
My goal is to publish from a MySQL db to Kafka. I've installed the mysql-connector jdbc (mysql-connector-java.noarch) and the jar file (mysql-connector-java.jar) is in the folder with the kafka connect jar file. I also installed jq, because Confluent kept badgering me to.
I am trying to create a custom connector to read from a remote database. Using the provided SQLite connector as a model (/etc/kafka-connect-jdbc/source-quickstart-sqlite.properties) I attempted to create my own connector file for my mysql database:
#mysql kafka publishing test
#
name=mysql-bulk-source
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=10
connection.url=jdbc:mysql://localhost:3306
mode=bulk
connection.user=kafkadbtest
connection.password=password
topic.prefix=mysql-
but when I run confluent load mysql-source -d /path/to/myfile/mysql-source.properties I get no output, and confluent status connectors is empty. Running confluent config mysql-connector -d /path/mysql-source.properties also does nothing, and following up with confluent config mysql-connector returns an empty config.
Confluent connect is up. I don't have any other ideas of what to try. Anyone can help? Any more info on my config that I should provide?
(p.s. there is no kafka tag, but it seems like there should be, and I lack the reputation to create it.)
centos mysql docker
asked Feb 6 at 21:06
Y e zY e z
1046
add a comment |
I am running a centOS docker container in an ec2 instance. I've installed Confluent, the necessary java files and placed them in the right places, and Kafka/ZooKeeper etc. are all up and running.
My goal is to publish from a MySQL db to Kafka. I've installed the mysql-connector jdbc (mysql-connector-java.noarch) and the jar file (mysql-connector-java.jar) is in the folder with the kafka connect jar file. I also installed jq, because Confluent kept badgering me to.
I am trying to create a custom connector to read from a remote database. Using the provided SQLite connector as a model (/etc/kafka-connect-jdbc/source-quickstart-sqlite.properties) I attempted to create my own connector file for my mysql database:
#mysql kafka publishing test
#
name=mysql-bulk-source
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=10
connection.url=jdbc:mysql://localhost:3306
mode=bulk
connection.user=kafkadbtest
connection.password=password
topic.prefix=mysql-
but when I run confluent load mysql-source -d /path/to/myfile/mysql-source.properties I get no output, and confluent status connectors is empty. Running confluent config mysql-connector -d /path/mysql-source.properties also does nothing, and following up with confluent config mysql-connector returns an empty config.
Confluent connect is up. I don't have any other ideas of what to try. Anyone can help? Any more info on my config that I should provide?
(p.s. there is no kafka tag, but it seems like there should be, and I lack the reputation to create it.)
centos mysql docker
asked Feb 6 at 21:06
Y e zY e z
1046
I am running a centOS docker container in an ec2 instance. I've installed Confluent, the necessary java files and placed them in the right places, and Kafka/ZooKeeper etc. are all up and running.
My goal is to publish from a MySQL db to Kafka. I've installed the mysql-connector jdbc (mysql-connector-java.noarch) and the jar file (mysql-connector-java.jar) is in the folder with the kafka connect jar file. I also installed jq, because Confluent kept badgering me to.
I am trying to create a custom connector to read from a remote database. Using the provided SQLite connector as a model (/etc/kafka-connect-jdbc/source-quickstart-sqlite.properties) I attempted to create my own connector file for my mysql database:
#mysql kafka publishing test
#
name=mysql-bulk-source
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=10
connection.url=jdbc:mysql://localhost:3306
mode=bulk
connection.user=kafkadbtest
connection.password=password
topic.prefix=mysql-
but when I run confluent load mysql-source -d /path/to/myfile/mysql-source.properties I get no output, and confluent status connectors is empty. Running confluent config mysql-connector -d /path/mysql-source.properties also does nothing, and following up with confluent config mysql-connector returns an empty config.
Confluent connect is up. I don't have any other ideas of what to try. Anyone can help? Any more info on my config that I should provide?
(p.s. there is no kafka tag, but it seems like there should be, and I lack the reputation to create it.)
centos mysql docker
centos mysql docker
asked Feb 6 at 21:06
Y e zY e z
1046
asked Feb 6 at 21:06
Y e zY e z
1046
edited Feb 6 at 21:18
Y e z
asked Feb 6 at 21:06
Y e zY e z
1046
asked Feb 6 at 21:06
Y e zY e z
1046
asked Feb 6 at 21:06
Y e zY e z
1046
1046
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1402863%2fcant-create-custom-kafka-connector%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1402863%2fcant-create-custom-kafka-connector%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
