pysparkshell works, but not python -> pyspark












0















I am having a problem with pyspark. I am able to run my cluster when I boot up pysparkshell from the command line aka:



pyspark --total-executor-cores 5 --executor-memory 3g


but when I run python and try to boot up the cluster I get the following issue:



ERROR TransportRequestHandler:193 - Error while invoking RpcHandler#receive() on RPC id 6381742667596359353

    java.io.InvalidClassException: org.apache.spark.storage.BlockManagerId; local class incompatible: stream classdesc serialVersionUID = 6155820641931972170, local class serialVersionUID = -3720498261147521052


anybody have experience with this? I cannot find another issue like this one online






bash python







share|improve this question



























    0















    I am having a problem with pyspark. I am able to run my cluster when I boot up pysparkshell from the command line aka:



    pyspark --total-executor-cores 5 --executor-memory 3g


    but when I run python and try to boot up the cluster I get the following issue:



    ERROR TransportRequestHandler:193 - Error while invoking RpcHandler#receive() on RPC id 6381742667596359353
    
    java.io.InvalidClassException: org.apache.spark.storage.BlockManagerId; local class incompatible: stream classdesc serialVersionUID = 6155820641931972170, local class serialVersionUID = -3720498261147521052


    anybody have experience with this? I cannot find another issue like this one online






    bash python







    share|improve this question

























      0












      0








      0








      I am having a problem with pyspark. I am able to run my cluster when I boot up pysparkshell from the command line aka:



      pyspark --total-executor-cores 5 --executor-memory 3g


      but when I run python and try to boot up the cluster I get the following issue:



      ERROR TransportRequestHandler:193 - Error while invoking RpcHandler#receive() on RPC id 6381742667596359353
      
    java.io.InvalidClassException: org.apache.spark.storage.BlockManagerId; local class incompatible: stream classdesc serialVersionUID = 6155820641931972170, local class serialVersionUID = -3720498261147521052


      anybody have experience with this? I cannot find another issue like this one online






      bash python







      share|improve this question














      I am having a problem with pyspark. I am able to run my cluster when I boot up pysparkshell from the command line aka:



      pyspark --total-executor-cores 5 --executor-memory 3g


      but when I run python and try to boot up the cluster I get the following issue:



      ERROR TransportRequestHandler:193 - Error while invoking RpcHandler#receive() on RPC id 6381742667596359353
      
    java.io.InvalidClassException: org.apache.spark.storage.BlockManagerId; local class incompatible: stream classdesc serialVersionUID = 6155820641931972170, local class serialVersionUID = -3720498261147521052


      anybody have experience with this? I cannot find another issue like this one online






      bash python




      bash python






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Feb 8 at 22:31









      planktonplankton

      12




      12






















          0






          active

          oldest

          votes











          Your Answer








          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "3"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1403697%2fpysparkshell-works-but-not-python-pyspark%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          0






          active

          oldest

          votes








          0






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes
















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Super User!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1403697%2fpysparkshell-works-but-not-python-pyspark%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          How to make a Squid Proxy server?

          Image
          1 I want to set up squid proxy server for monitoring user activity in a network and Internet Web URL filtering for user. I also want to configure the user login, user log, user access group, and block unwanted web sites in my network. Please help me and guide me the full process to do this. server proxy squid share | improve this question edited Jul 31 '12 at 14:45 Jorge Castro 37.1k 107 422 617 asked Jul 31 '12 at 5:16 Goivind Tiwari Goivind Tiwari ...
          Read more

          Is this a new Fibonacci Identity?

          Image
          6 1 $begingroup$ I have found the following Fibonacci Identity (and proved it). If $F_n$ denotes the nth Fibonacci Number, we have the following identity begin{equation} F_{n-r+h}F_{n+k+g+1} - F_{n-r+g}F_{n+k+h+1} = (-1)^{n+r+h+1} F_{g-h}F_{k+r+1} end{equation} where $F_1 = F_2 = 1$ , $r leq n$ , $h leq g$ , and $n, g, k in mathbb{N}$ . It is not too hard to show that this identity subsumes Cassini's Identity, Catalan's Identity, Vajda's Idenity, and d'Ocagne's identity to name a few. I have done a pretty thorough literature review, and I have not found anything like this, but I am still wondering if anyone has seen this identity before? I found this by accident after noticing some patterns in some analysis work I was doing, so if this is already known I would be curious to see w...
          Read more

          19世紀

          Image
          この記事は検証可能な参考文献や出典が全く示されていないか、不十分です。 出典を追加して記事の信頼性向上にご協力ください。 （ 2012年3月 ） 千年紀: 2千年紀 世紀: 18世紀 - 19世紀 - 20世紀 十年紀: 1800年代 1810年代 1820年代 1830年代 1840年代 1850年代 1860年代 1870年代 1880年代 1890年代 19世紀に君臨した大英帝国。 19世紀 （じゅうきゅうせいき）は、西暦1801年から西暦1900年までの100年間を指す世紀。 目次 1 19世紀の歴史 1.1 国民国家の成立 1.2 帝国主義の興隆 1.3 列強の植民地争奪戦 2 できごと 2.1 1800年代 2.2 1810年代 2.3 1820年代 2.4 1830年代 2.5 1840年代 2.6 1850年代 2.7 1860年代 2.8 1870年代 2.9 1880年代 2.10 1890年代 2.11 1900年代 3 文化 3.1 文学 3.2 音楽 3.3 思想 3.4 科学 3.5 技術 4 人物 4.1 ヨーロッパ 4.1.1 政治家・王族 4.1.1.1 フランス 4.1.1.2 オーストリア＝ハンガリー 4.1.1.3 ロシア 4.1.1.4 イギリス 4.1.1.5 ドイツ（プロイセンほかドイツ領邦を含む） 4.1.1.6 北欧 4.1.1.7 イタリア 4.1.1.8 スペイン 4.1.1.9 ベルギー 4.1.1.10 ギリシア 4.1.2 軍人 4.1.3 実業家 4.1.4 科学と技術 4.1.5 思想と哲学・人文諸学 4.1.6 宗教 4.1.7 文学 4.1.8 美術 4.1.9 音楽 4.1.10 社会事業家 4.1.11 探検家・旅行家 4.1.12 料...
          Read more